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This  study  evaluates  the  results , uses , and  benefits  of  a sample 
of  Army  studies  completed  or  terminated  during  FY  74  and  FY  75.  The 
primary  concern  is  with  study  evaluation,  the  extent  to  which  individ- 
ual studies  achieve  stated  goals  and  objectives.  (The  question  of 
whether  those  stated  goals  and  objectives  are  appropriate — considered 
a part  of  total  program  evaluation — is  not  addressed  in  this  study.) 

Data  were  gathered  from  questionnaires  completed  by  study  sponsors  and 
users  and  from  examination  of  study  documents.  Included  in  the  sample 
are  studies  in  all  The  Army  Study  System  (TAS5)  categories  and  by  all 
means  of  accomplishment.  Results  are  in  the  form  of  observations,  con- 
clusions, and  recommendations  related  to  resource  levels,  study  achieve- 
ments, use  and  nonuse,  benefits,  critical  issues  for  study,  and  TASS. 
These  results  are  intended  to  be  used  by  top  study  management  when  taking 
actions  or  making  decisions  regarding  funding  levels , program  focus , 
study  undertakings,  and  improvements  in  study  planning,  programing,  and 
execution  to  enhance  chances  of  study  success  and  usefulness.  Not 
included  in  this  report  is  a paper  on  study  program  management  which 
was  provided  to  thi  study  sponsor  under  separate  cover. 
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SUMMARY 


1.  The  Army  Study  System  (TASS)  enables  the  Army  to  respond  to 
changing  conditions  based  on  thorough  analyses  of  current  and  pro,-’ acted 
problems.  Studies  are  basic  to  the  formulation  of  concepts,  doctrine, 
plans,  and  policies.  They  help  high-level  managers  in  making  decisions 
and/or  providing  persuasive  inputs  to  the  Joint  Staff  and  the  Department 
of  Defense. 

2.  Army  studies  are  characterized  by  decentralized  program  develop- 
ment and  centralized  review  and  evaluation.  They  are  conducted  by  a 
wide  variety  of  in-house  organizations  and  by  contract  support.  Manage- 
ment responsibilities  are  delegated  to  Army  Staff  agencies  and  major 

Army  commands  (MACOMs) . Although  HQDA  studies  were  evaluated  (for  results 
and  use)  in  1974  and  HQDA  and  MACOM  studies  were  evaluated  in  1975.  con- 
tinued reduction  of  resources  available  for  studies  indicated  that  a 
more  detailed  assessment  was  needed  at  this  time.  The  Director  of 
Management,  Office,  Chief  of  Staff.  Army,  therefore.,  requested  that  ESG 
undertake  this  analysis. 

3.  The  objectives  of  this  study  are  to: 

a.  Determine  the  immediate,  residual,  and  derivative  impacts 
of  studies  evaluated.  Evaluate  the  cost  effectiveness  of  individual 
studies  that  have  resulted  in  tangible  benefits. 

b.  Identify  common  characteristics/attributes  of  successful 
studies.  Identify  common  causes  for  study  failure.  Draw  inferences 
from  success  and  failure  and  make  recommendations  regarding  study  efforts 
and  resources. 

c.  Affirm  the  validity  and  usage  of  available  data  bases  (e.g.. 
Defense  Documentation  Center,  Defense  Logistics  Studies  Information 
Exchange,  and  the  Army  Study  Documentation  and  Information  Retrieval 
Sytems)  during  the  development  and  execution  of  study  efforts.  Review 
existing  procedures  to  determine  if  study  information  is  being  dissemi- 
nated adequately  to  preclude  duplicating  study  effort. 

d.  Contribute  to  a better  understanding  of  the  relative  cost 
effectiveness  of  in-house  and  contract  studies.  Obtain  insights  into 
the  range  of  resource  levels  the  Army  should  commit  to  and  within  the 
study  program. 

e.  Develop  criteria  for  use  in  evaluating  the  merits  of  study 
proposals. 
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4.  This  study  analysis  is  based  on  data  pertaining  to  the  145 
studies  chosen  as  a sample  from  462  studies  completed  or  terminated  in 
FY  74  and  FY  75.  The  sample  represented  a resource  investment  of  about 
$50  million,  both  in-house  and  contract,  or  approximately  one-half  the 
resources  consumed  by  the  462  total.  The  study  data  were  acquired  by 
mailed  questionnaires,  by  examing  study  reports  and  other  documents,  and 
by  follow-on  interviews  by  ESG  analysts.  A principal  purpose  of  the 
interviews  was  to  verify  the  reliability  of  questionnaire  responses. 

ESG  analyzed  the  acquired  data  on  each  individual  study  and  the  total 
consolidated  sample  and  arrived  at  the  following  findings  and  conclusions. 

a.  There  is  a tendency  at  all  levels  to  equate  a study's 
importance  to  the  hierarchical  position  of  the  requestor.  There  is 

no  generally  used  mecnanism  which  establishes  priorities  based  on  goals, 
substance,  or  issues  addressed. 

b.  Critical  Army  issues  are  identified  and  ordered  in  only 
enough  detail  to  influence  planned  resource  allocations  in  very  broad 
terms.  Allocating  resources  based  on  level  of  effort  per  study  category 
is  meaningless  because  the  categories  are  not  strictly  defined. 

c.  Return  on  study  resource  investment  can  be  improved  by 
critically  weighing  certain  types  of  study  efforts  and  by  strengthening 
study  management  from  inception  to  implementation.  Results  and  use 
data  do  not  indicate  imbalances  between  in-house  and  contract  efforts. 

d.  Study  Planning  Guidance  (SPG)  has  marginal  causal  effect 
on  the  focus  and  content  of  the  annual  study  programs.  There  is  no 
direct  connection  between  SPG  and  appropriate  levels  of  effort. 

e.  Studies  generally  achieve  most  of  what  they  set  out  to  do; 
however,  this  is  not  a guarantee  that  the  study  results  will  be  effec- 
tively used. 

f.  Causes  for  not  using  study  results  include:  turbulence  in 

study  management,  insufficient  data,  delays  in  approval,  delay  in  comple- 
tion/problam  passed  with  time,  and  insufficient  knowledge  as  to  the 
study's  usefulness  either  because  it  was  not  publicized  or  because  docu- 
mentation did  not  permit  confident  use  of  the  results. 

g.  It  is  difficult  (and  maybe  meaningless)  to  compute  the 
cost  effectiveness  of  studies  because  the  assumptions  drive  the  solution 
of  such  calculations. 
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h.  There  appears  to  be  no  doubt  that  TASS  produces  significant 
benefits.  Ultimate  benefits  are  not  explicitly  considered  when  a study 
is  undertaken. 

i.  Except  at  HQDA  level,  existing  study  data  are  not  widely 
used  for  the  purposes  intended.  Data  discipline  needs  to  be  enforced  in 
the  preparation  of  study  summaries  (DD  Forms  1498);  entries  pertaining 
to  results  and  implementation  are  proving  particularly  inadequate. 

5.  TASS  has  experienced  a rather  constant  and  evolutionary  change 
over  the  years.  The  following  recommendations  are  intended  to  give  top 
study  management  a basis  for  directing  future  TASS  improvements. 

a.  Do  not  now  radically  change  the  manner  in  which  individual 
study  programs  are  developed  but  revamp  the  SPG  to  reflect  key  xssues  and 
to  distinguish  between  important  and  less  important  priority  areas. 

b.  Study  proposals  should  be  evaluated  to  insure  that: 

(1)  The  problem  and  purpose  are  clearly  defined. 

(2)  The  need,  expected  results,  and  user  are  identified. 

(3)  There  are  a manageable  number  of  objectives  combined 
with  consistent  scope  and  resource  estimate. 

(4)  There  is  assurance  of  high-level  sponsor  interest  and  a 
specified  sponsor  management  mechanism  (i.e.,  steering/advising  group  or 
manager . ) 


c.  The  problem  of  action  officer  and  study  advisory  group 
turbulence  needs  to  be  resolved.  Top  management  should,  as  a matter  of 
policy,  be  sure  that  individuals  assigned  to  these  positions  expect  to 
be  available  for  the  duration  of  the  study. 

d.  Executives  and  commanders  need  to  become  more  involved 
early — when  the  need  for  a study  is  established.  It  may  be  advisable 
to  establish  a threshold  above  which  they  must  personally  approve  study 
undertakings . 


e.  Results,  uses,  and  benefits  data  must  be  recorded  for 
future  program  evaluation  purposes  before  all  people  involved  in  the 
study  execution  and  implementation  have  departed.  The  implementation 
plan  should  have  a provision  for  gathering  information  necessary  to  a 
comprehensive  and  valid  appraisal  of  results  and  use. 
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f.  Publish,  at.  least  quarterly,  a list  of  all  studies  completed 
or  terminated  during  the  period. 

g.  Institute  a simple  newsletter  disseminating  items  of  inter- 
est to  the  study  community. 

h.  Resource  levels  should  be  keyed  to  actual  identified  need 
rather  than  arbitrary  level  of  effort.  Cutbacks  in  the  short  run  can 
be  absorbed  with  least  loss  of  benefits  by: 

(1)  Curtailing  large-scale  model  and  simulation  efforts. 

(2)  Curtailing  broad  methods  and  standardization  projects. 

(3)  Selectively  reducing  contract  support. 

i.  AR  5-5,  which  defines  TASS  and  prescribes  administrative 
procedures,  must  be  strongly  enforced.  This  is  essential  if  study  top 
management  is  to  exert  control  over  the  system. 

6.  This  report  is  concerned  primarily  with  evaluating  results, 
use,  and  benefits  of  individual  studies.  ESG  has  supplied  the  Director 
of  Management  a separate,  informal  paper  which  relates  to  total  program 
evaluation. 
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RESULTS  AND  USE  OF  ARMY  STUDIES 


I.  GENERAL 

1.  Purpose,  This  study  assesses  the  results,  uses,  and  benefits  of 
Army  studies  to  be  considered  by  the  Director  of  Management,  Office  of 
the  Chief  of  Staff  of  the  Army  (OCSA) , in  decisions  and  actions  regarding 
the  directions  of  current  and  future  Army  study  efforts. 

2.  Scope. 

a.  This  report  is  based  on  analysis  of  a 145-study  sample 
selected  from  over  460  studies  completed  or  terminated  during  FY  74  and 
FY  75.  In-house  and  contractor  studies  in  all  The  Army  Study  Program 
(TASP)  categories  have  been  included. 

b.  The  evaluation  of  each  study  did  not  include  a peer  review 
of  study  contents  or  procedure  or  any  attempt  to  duplicate  the  results. 
Rather,  the  analysis  concentrated  on  results  of  the  studies  in  terms  of 
the  objectives  and  the  uses  made  of  these  results.  Mailed  questionnaires 
served  as  the  primary  data-collection  instrument. 

c.  Both  tangible  and  intangible  benefits  are  assessed  relative 
to  study  purposes,  issues  addressed,  achievements,  and  uses  in  the  limited 
context  of  individual  study  evaluation.  The  much  broader  question  of 
whether  individual  study  goals  and  objectives  are  appropriate  is  consid- 
ered a part  of  total  program  evaluation  and  is  given  much  less  attention 
in  this  study.  (Howevei  , insights  related  to  program  evaluation  have 
been  supplied  to  the  Director  of  Management  in  a separate,  informal  paper.) 


d.  Administrative  procedures  and  regulations  governing  The  Amy 
Study  System  (TASS)  are  discussed  to  the  extent  that  they  influence  the 
studies'  results,  uses,  and  benefits. 

3.  Background. 

a.  TASS  is  defined  in  AR  5-5.  It  enables  the  Army  to  respond 
to  changing  conditions  based  on  thorough  and  detailed  analyses  of  present 
and  projected  problems.  Army  studies  are  characterized  by  decentralized 
program  development  and  centralized  review  and  evaluation.  Management 
responsibilities  are  delegated  to  Army  Staff  agencies  and  major  Army 
commands  (MACOMs) . Procedures  emphasize  quality  control  and  efficient 
allocation  of  study  resources.  A primary  TASS  objective  is  to  conduct 
studies  that  will  produce  usable  results.  Those  results  should  aid 
decisionmaking  by  presenting  alternative  solutions  to  problems  and/or 
providing  a basis  for  Army  inputs  to  Joint  Staff  and  Department  of  Defense 
(DOD)  policies  and  positions. 

b.  TASS  evolution  over  the  years  has  been  directly  influenced 

1/2/3/ 

by  several  critical  assessments.— These  efforts  concentrated 

primarily  on  management  and  administrative  procedures  within  the  system. 

There  has  been  no  comprehensive  effort  to  address  the  application  of  study 

results  nor  to  identify  benefits  resulting  from  study  resource  invest- 
4/ 

ments .— 

1/  DA,  OAVCSA,  After-Action  Report  of  PRIMAR. 

2 J DA,  OAVCSA,  Final  Report  of  the  Committee  to  Evaluate  the  Army 
Study  System  (ETASS) . 

3/  DA,  Ofc  of  the  VCSA,  The  Army  Study  System  (Bonesteel  Report) . 

4/  There  have  been  studies  in  this  area  with  limited  scope  concen- 
trating primarily  on  models,  simulations,  and  war  games.  Three  of  the 
more  recent  efforts  are  listed  as  references  8,  9,  and  10. 
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c.  The  Director  of  Management,  OCSA,  was  concerned  that  scarce 
study  resources  must  be  effectively  committed  and  noted  the  need  for  a 
definitive  review  of  study  results,  uses,  and  benefits.  The  Engineer 
Studies  Group  (ESG)  was  tasked  to  perform  that  review.Ji/  This  report 
contains  the  results  of  that  review. 

4.  Organization  of  the  Report.  This  main  paper  is  summary  in  nature. 
Major  findings  and  conclusions  are  discussed  in  Part  II  and  recommendations 
are  developed  in  Part  III. 

a.  Annex  A presents  a more  detailed  discussion  of  the  review 
process  (sample  selection,  data  gathering,  verification,  analysis,  syn- 
thesis) . It  also  contains  the  data  tabulations  and  consolidated  responses 
to  the  questionnaires. 

b.  Annex  B contains  a list  of  the  studies  reviewed  and  Annex  C 
contains  the  Bibliography. 

II.  MAJOR  FINDINGS  AND  CONCLUSIONS 

5.  General . Overall,  the  TASS  appears  to  be  in  better  administrative 
health  than  its  critics  have  led  us  to  believe.  Within  agency/command 
programs,  customer  satisfaction  and  demand  for  studies  are  high.  Studies 
and  analyses  were  found  to  be  initiated  and  accomplished  in  line  with  appli- 
cable directives  and  regulations.  However,  there  appear  to  be  opportunities 
for  substantial  improvement.  The  study  system  itself  appears  to  be  in  a 
continuous  state  of  change.  Even  as  this  study  was  in  progress,  revisions 

5 J DA,  Ofc  of  the  AGf  Ltr,  Study:  Results  and  Use  of  Army  Studies. 
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to  study  regulations  were  being  staffed  and  administrative  changes  were 


being  made  anticipating  approval  of  the  revisions.  Study  program  managers, 


particularly  at  staff  agency/MACOM  level  appear  to  have  reasonable  control 


of  the  admittedly  imperfect  study  programs.  It  is  apparent  that  the 


reports  and  approval  procedures  embodied  in  TASS  have  resulted  in  more 


thought  and  care  being  given  to  allocation  of  study  resources.  The  issues 


addressed  by  the  hundreds  of  studies  always  underway  range  from  very 


narrow  functional  problems  to  the  most  complex  planning  and  policy  issues 


affecting  the  entire  Army  and  DOD.  Decisions  at  all  levels  are  generally 


complex.  Specific  decisions  are  a result  of  many  compromises  among 


competing  considerations.  It  is  difficult — and  will  remain  difficult — to 


single  out  a specific  decision  and  say  "that  one  was  different  because  of 


a study." 


6.  Study  Requirements.  Based  on  the  premise  that  originators  of 


study  requirements  would  be  the  most  likely  users  and  beneficiaries  of 


study  results,  the  sources  of  study  requirements  were  identified.  Figure  1 


shows  the  percentages  of  DA  Staff-  and  MACOM-sponsored  study  programs  that 


originated  at  the  various  levels  and  similar  percentages  for  the  total 


TASP.  (Unless  otherwise  noted,  data  displays  in  this  main  paper  are  based 


on  the  questionnaire  responses  pertaining  to  141  studies.  Adequate  infor- 


mation was  not  available  for  four  of  the  145  studies  in  the  sample.  Annex 


A contains  the  details.) 
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ORIGINS  OF  STUDY  REQUIREMENTS 


DA  Staff 

MACOM 

Total 

TASP 

Originator 

% 

Number 

% 

Number 

% 

Number 

DOD/JCS 

10.7 

8 

4.6 

3 

7.8 

11 

CSA/SA 

16.0 

12 

13.6 

9 

14.9 

21 

Staff  Agency 

66.6 

50 

27.3 

18 

48.3 

68 

MACOM 

1.3 

1 

48.5 

32 

23.4 

33 

Other 

2.7 

2 

3.0 

2 

2.8 

4 

Unsolicited 

Proposal 

2.7 

_2 

3.0 

_2 

2.8 

4 

Total 

100.0 

75 

100.0 

66 

100.0 

141 

Figure  1 

a.  Approximately  26  percent  of  the  DA  Staff  Agency  (hereafter 


referred  to  as  DA  Staff)  and  45  percent  of  the  MACOM  study  programs 
originate  from  above.  This  finding  is  significant  but  not  surprising  since 
it  was  also  found  that  the  DA  Staff  and  MACOMs  invariably  give  highest 
priority  to  requests  from  higher  echelons.  Most  DA  Staff  and  MACOMs  have 
procedures  for  assigning  priorities  and  resolving  conflicts  among  the 
studies  they  originate.  Externally  originated  study  requirements  receive 
priority  treatment  and  consequently  were  found  to  be  the  most  frequent 
cause  for  deferring/terminating  studies  in  the  DA  Staff /MACOM  annual  study 
programs.  Except  for  those  instances  where  the  externally  directed  study  is 
tied  in  with  a recurring  Planning,  Programing,  and  Budgeting  System  (PPBS) 


or  similar  requirement,  it  is  difficulc  for  the  ultimate  DA  Staff/MACOM 
sponsor  to  anticipate  and  program  for  the  study. 

b.  Conclusion.  There  is  a tendency  at  all  levels  to  equate 
a study's  importance  to  the  hierarchial  position  of  the  requestor.  There 
is  no  generally  used  mechanism  which  establishes  priorities  regardless  of 
study  origin  and  based  on  program  goals,  substance,  or  issues  addressed. 

7.  Resource  Levels.  Figure  2 sho\>s  the  resources  committed  to  the 
studies  completed  or  terminated  during  FY  74  and  FY  75  and  that  portion 
consumed  by  the  studies  in  the  sample.  (Costs  are  cumulative  and  cannot 
be  interpreted  as  covering  any  one  fiscal  year.)  Figure  3 indicates  how 
the  studies  were  distributed  among  the  categories  and  means  of  accomplish- 
ment. The  data  show  that  in-house  efforts  accounted  for  2/3  of  the  studies 
and  less  than  2/3  of  the  total  costs.  ESG  pursued  the  analysis  of  resource 
levels  along  the  lines  suggested  by  the  categories  in  Figures  2 and  3. 

a.  Relative  cost  effectiveness  is  not  clear  since  in-house  and 
contract  resources  are  often  applied  to  the  same  study  in  a collaborative 
effort.  Further,  no  obvious  duplication  of  effort  (i.e.,  both  in-house 
and  contract  resources  committed  to  the  same  complete  problem  independently) 
made  comparisons  easier.  Their  more  stringent  accounting  procedures  make 
contract  expenditures  easier  to  determine  than  in-house  expenditures. 
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Benefits,  on  the  other  hand,  are  more  easily  identified  for  in-house 
studies  due  to  the  high  density  of  functional  issues  addressed.  Figure  4 
shows  the  average  costs  of  in-house  and  contract  studies. 


COSTS  FOR  STUDIES 

COMPLETED/TERMINATED  IN  FY  74  AND  FY  75 


Costs 


Number 

In-house 

Contract 

Totals/ 

of  Studies 

(PMY) 

($K) 

($K) 

Included  in  Evaluation 

145 

626 

19,527 

50,300 

Total  During  Period 

462 

1,363 

38,950 

107,100 

a / Calculated  using  $50K  per  in-house  professional  man-year.  Use 
of  $50K  per  PMY  is  somewhat  arbitrary.  DOD  used  that  amount  for  esti- 
mates in  1974.  It  is  not  a DA-approved  planning  factor.  A detailed 
cost  analysis  would  likely  find  that  in-house  costs  are  lower,  possibly 
as  low  as  $40K  per  PMY.  Contract  costs,  on  the  other  hand,  do  not  in- 
clude the  in-house  costs  for  contract  administration  (i.e.,  contracting 
officer,  advisory  group,  and  other  necessary  management).  Refined 
estimates  would  tend  to  tip  the  cost  balance  in  favor  of  in-house 
studies. 


Figure  2 

b.  The  study  categories  are  not  particularly  useful  as  indicators 
of  resource  imbalances.  They  do,  however,  focus  attention  on  the  general 
content  of  the  matter  being  studied.  Except  for  those  very  narrow  in 
scope,  studies  frequently  can  be  assigned  to  more  than  one  category. 


THE  "AVERAGE"  STUDY  COST 
(FY  74  and  FY  75) 


In-house 

Contract 

Combined 

Number 

Category 

Man- 

years 

Dollars 

(K>£/ 

Dollars 

(K) 

Average 

Cost 

of 

Studies 

1 

7.48 

374.0 

328.5 

355.9 

33 

2 

3.55 

177.3 

205.3 

187.7 

78 

3 

5.86 

293.1 

303.4 

297.2 

107 

4 

4.72 

236.2 

292.2 

248.7 

112 

5 

2.44 

122.0 

177.8 

143.1 

106 

6 

4.87 

243.4 

179.7 

226.3 

26 

Overall 

4.47 

223.5 

248.0 

231.8 

462 

a / One  man-year  = $50K. 


Figure  4 

c.  The  question  of  resource  imbalances  remains  unanswered.  ESG's 
analysis  did  not  disclose  any  reasons  to  suggest  that  significant  reallo- 
cations should  be  made  based  on  the  FY  74  and  FY  75  data.  It  was  found 
that  quick-reaction  study  requirements  are  rather  effectively  met  by 
deferral  of  ongoing,  less  urgent  efforts  and  occasional  creation  of  ad 
hoc  study  groups.  Organizations  with  an  in-house  study  capability  are 
decidedly  more  responsive  to  quick-reaction  studies  than  are  contractors, 
particularly  since  the  demise  of  Army-affiliated  Federal  Contract  Research 
Centers  (FCRCs)  and  level-of-ef fort  contracts. 
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d.  The  Array  is  fundamentally  concerned  that  its  study  resources 
should  be  applied  to  genuine  Army  problems.  TASS  was  developed  to  give 
HQ  DA  a broader  view  of  its  study  program  and  greater  assurance  that  its 
TASP  funds  are  being  spent  wisely.  From  a sponsor's  viewpoint,  the  problems 
being  studied  are  genuine  and  must  be  solved  if  the  Army's  missions  and 
functions  are  to  be  executed  effectively.  In  this  sense,  resource  commit- 
ments must  be  justified  from  an  operational  standpoint --as  part  of  the  cost 
of  doing  the  job  we  have  to  do.  In  effect  all  problems  would  have  been 
analyzed  regardless  of  whether  the  results  of  analysis  are  termed  as  a 
"study."  The  net  result  is  that  individual  program  sponsors  will  commit 
resources  to  DA  Staff/MACOM  mission  and  function  problems  (which  are  pri- 
oritized by  the  sponsor)  independent  of  other  programs.  HQ  DA  combines 
these  individual  programs  to  form  TASP.  TASP  is  not  constructed  to  iden- 
tify which  studies  are  most  important  (overall  to  the  Army)  in  terms  of 
critical  issues /problems /decisions  addressed. 

e.  Roughly  one-third  of  the  studies  in  the  sample  were  classed 
as  having  a low  rate  of  return  on  the  resource  investment.  These  studies 
had  limited  use,  marginal  impact,  and  low  or  unrecognizable  benefits.  This 
classification  is  a judgment  based  on  questionnaire  responses  identifying 
benefits,  uses,  and  impacts  of  each  study.  All  of  the  questions  were 
relative  to  the  original  study  purpose  and  objectives.  Studies  in  this 
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low-return  class  were  spread  over  all  categories  and  means  of  accomplish- 
ment. However,  there  were  some  characteristics  which  could  be  useful  as 
caution  signals  in  future  undertakings. 

(1)  Large-scale  raodel/method/standardization  developments 
are  very  expensive  and  risky  ventures.  Although  the  theory  or  process 
studied  may  .lave  been  developed,  several  of  these  undertakings  were  poor 
investments  (particularly  in  the  short  run)  and  were  not  used  because 
real  data  were  not  available.  Most  are  salvageable  if  one  is  willing  to 
commit  additional  resources  to  data  development.  In  other  instances, 
particularly  in  methods/slandardization  projects,  the  underlying  object 
of  study  was  nonstable  or  ill  defined.  Resultant  methods/standardization 
developments  tend  to  be  overtaken  by  events. 

(2)  Low  return  on  study  resource  investment  is  associated 
with  discontinuities  in  study  management  (changes  in  action  officers, 
sponsor  representatives,  SAG  principals,  or  in  the  study  agency).  Their 
differing  perceptions  of  the  original  problem  and  anticipated  uses  caused 
ongoing  studies  to  flounder  and  even  successful  studies  (all  objectives 
met)  to  be  less  than  fully  used. 

(3)  A very  long  or  phased  study  was  found  to  be  a prime 
candidate  for  a low-return  rating.  These  are  likely  to  be  large-scale 
developments  ((1)  above),  vulnerable  to  (2)  above,  are  expensive,  and 
the  ultimate  costs  are  generally  underestimated.  These  studies  also 
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have  little  direct  value  when  terminated  prior  to  completion — an  all  or 
none  situation. 

f.  Conclusions: 

(1)  Allocating  resources  based  on  level  of  effort  per 
category  could  be  meaningless  if  study  categories  are  not  strictly 
defined. 

(2)  Critical  Array  issues  are  identified  and  ordered  in  only 
enough  detail  to  influence  planned  resource  allocations  in  broad  terms. 

(3)  Results  and  use  data  do  not  suggest  imbalances  between 
in-house  and  contract  efforts. 

(4)  Return  on  study  resource  investment  can  be  improved  by 
critically  weighing  certain  types  of  study  efforts  and  by  strengthening 
study  management  from  inception  to  implementation. 

8.  Program  Development  Study  Planning  Guidance  (SPG)  is  designed 
as  the  vehicle  which:  knits  DA  Staff  and  MACOM  study  programs  into  the 

TASP;  provides  a benchmark  for  HQ  DA  evaluation,  review,  or  approval  of 
the  study  programs;  and  helps  in  establishing  appropriate  levels  of  effort 
for  priority  area  studies.  ESG  explored  program  development  to  determine 
the  extent  to  which  anticipated  study  results,  uses,  and  benefits  are 
considered  and  how  critical  (priority)  Army  issues  are  identified  when 
programing  study  resources. 
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a.  The  most  important  factor  in  determining  DA  Staff  or  MACOM 
study  program  content  was  found  to  be  the  DA  Staff/MACOK  perception  of 
its  mission  and  functions.  Due  to  the  way  SPG  is  developed,  local  pro- 
grams are  essentially  feeding  the  DA  guidance  instead  of  vice  versa. 

Unless  the  study  origin  is  external  to  the  DA  Staff/MACOM,  critical 
issues  are  really  local  functional/mission  issues. 

b.  It  is  relatively  easy  to  identify  and  justify  studies  with 
specific  SPG  areas  of  concern  (not  surprisii  g in  light  of  the  way  SPG 
is  developed).  Therefore,  SPG  ic  not  ac  effective  as  it  could  be  because 
it  incorporatts  so  many  areas,  there  is  no  scale  of  in -ortance  or  priority 
within  the  areas  of  concern,  and  there  is  no  mechanism  to  resolve  conflicts. 

c.  Conclusion.  The  SPG  has  marginal  causal  effect  on  the  focus 
and  content  of  the  annual  study  programs.  TASP  is  much  more  a hetero- 
geneous conglomerate  than  a homogeneous  program.  There  is  no  direct 
connection  between  SPG  and  appropriate  levels  of  effort. 

9.  Study  Results.  Figure  5 shows  the  extent  to  which  sponsors, 
users,  and  study  agencies  feel  that  study  objectives  were  met.  Question- 
naire respondents  were  very  confident  of  their  assessments.  Although 
ESG's  followup  verification  of  34  studies  and  examination  of  141  DD  1498’ s 
tended  to  confirm  respondents'  opinions,  some  credit  for  achievement  may 
have  been  overly  generous. 


Figure  5 


a.  Although  85  percent  of  the  studies  achieve  most  or  all  of 
their  objectives,  as  many  as  45  percent  fail  to  achieve  one  or  more  of 
their  objectives.  The  most  frequent  causes  of  failure  to  meet  objec- 
tives were:  (in  descending  order) 

(1)  Too  many  objectives.  We  call  this  the  shotgun  approach. 
Multiple  objectives  often  created  conflicts  which  were  solved  by  ignoring 
one  or  more  of  the  conflicting  objectives.  This  cause  of  failure  is 
compounded  by  an  almost  universal  underestimation  of  resources  required 

to  address  all  objectives  adequately  within  the  allotted  time. 

(2)  Trying  to  structure  a nonstable,  poorly  defined  activity. 
This  is  the  primary  cause  of  failure  to  achieve  objectives  in  studies 
which  address  methods,  systems,  and  model  improvement /development. 


(3)  Reorganization.  Some  of  the  studies  evaluated  were 
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in  process  during  the  CONUS  and  DA  Staff  reorganization.  Resultant 
changes  in  roles,  missions,  and  functions  and  dissolution  of  sponsoring 
elements  were  cited  as  reasons  objectives  were  not  met. 

(4)  Change  within  sponsoring  organization  or  study  agency. 
Change  within  sponsor  organization  causes  failure  primarily  because  new 
people  have  different  perceptions  of  the  problem  being  studied  or  the  need 
for  the  study.  Within  the  study  agency,  failure  is  due  primarily  to  loss 
of  expertise/capability  by  promotion,  transfer,  retirement,  or  reduction 

in  budget  or  manning  levels.  (This  cause  is  somewhat  related  to  (3)  above.) 

(5)  Methodology  incapable  of  solving  the  problem.  This 
cause  is  usually  related  to  overestimating  the  tractability  of  the  prob- 
lem and  the  robustness  of  current  analytic  tools.  (Related  somewhat  to 
(2)  above.) 

b.  ESG  found  that  even  though  a study  is  classed  as  meeting  one 
or  more  of  its  objectives  (132  studies),  it  may  not  have  had  a widely 
recognized,  significant  impact.  Significant  impact  is  interpreted  as  that 
which  occurs  either  because  the  study  results  are  used  by  one  or  more 
elements  external  to  the  sponsoring  element  or  the  study  results  were 
essential  inputs  to  a decision  or  problem  solution.  Figure  6 shows  that, 
on  this  basis,  about  55  percent  of  the  studies  had  significant  impact. 
Studies  in  the  low-impact  category  frequently  are  extremely  important 
in  a very  narrow  application. 


15 


. 


w 


U 

LI 


1 1 
i.  -j 


t 


i 


IMPACT  OF  STUDIES 


Objectives 

Significant 

Marginal 

Ncne 

Unknown 

Total 

None  Met 

0 

0 

9 

0 

9 

Most  Not  Met 

1 

6 

4 

2 

13 

Most  Met 

25 

11 

3 

3 

42 

All  Met 

52 

18 

3 

4 

77 

Total 

78(55%) 

35(25%) 

19(14%) 

9(6%) 

141(100%) 

Figure  6 


c.  Conclusion.  Studies  generally  achieve  most  of  what  they  set 
out  to  do.  However,  earlier  perceptions  of  TASP's  overall  impact  on  the 
Army  have  probably  been  clouded  because  routine  problems  of  important 
but  local  concern  to  a DA  Staff  agency  or  MACOM  are  part  of  TASP.  Most 
local  concerns  may  not  have  Army-wide  importance. 

10.  Uses  of  Study  Results.  Figure  7 shows  that  of  the  132  studies 
which  met  one  or  more  objectives,  all  except  10  (8  percent)  were  used  to 
some  degree  and  about  half  of  these  provided  direct  input  to  a decision 
or  problem  solution.  Eighteen  percent  were  used  as  input  to  plans  or 
planning  processes  such  as  JSOP.  The  15  percent  which  were  used  as  input 
to  another  study  or  were  a phase  of  a larger  study  generally  have  little 
stand-alone  use  or  value.  Studies  whose  results  are  used  primarily  as 
reference  data  or  planning  factors  (14  percent)  are  generally  undertaken 
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with  no  particular  decision  in  mind  although  they  may  ultimately  influence 
many  decisions.  Reasons  for  nonuse  of  10  studies  which  achieved  one  or 
more  objectives  were  investigated  further. 


USE  OF  STUDIES  WHICH  ACHIEVED 
ONE  OR  MORE  OBJECTIVES 


Type  Uses*/ 

Percent 

Number 

Direct  Input  to 
Decision/Problem 

45 

59 

Input  to  a 

Planning  Process 

18 

24 

Input  to  Another 
Study  (Or  Phase) 

15 

20 

Reference  Data  and 
Planning  Factors 

14 

19 

Not  Used 

8 

10 

Total 

100 

132 

a./  Most  studies  have  more  than  one  possi- 
ble use.  They  were  classed  according  to  the 
use  which  most  closely  matched  the  original 
purpose  for  undertaking  the  study.  The  nine 
studies  which  achieved  no  objectives  are  not 
included  in  this  figure. 


Figure  7 


a.  ESG  found  that  nonuse  was  most  frequently  related  to  changes 
in  the  personalities  involved  (i.e.  , changes  in:  action  officers,  SAG 

principals,  chiefs  of  agencies,  and  commanders).  Insufficient  data  was 
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a close  second,  but  this  cause  of  nonuse  primarily  affected  methods  and 
model-related  studies.  Other  causes  for  nonuse  were: 

Cl)  Authorities'  lack/delay  in  approving  study  results. 

C2)  The  problem  or  decision  which  the  study  addressed 
was  overtaken  by  events. 

(3)  Potential  users  questioned  the  logic  underlying  the 
study  and  its  results. 

(4)  Lack  of  knowledge  of  the  study's  usefulness  either 
because  it  was  not  publicized  or  the  documentation  was  insufficient  to 
permit  confident  use  of  the  results. 

b.  Conclusion.  Achieving  objectives  is  not  a guarantee  that 
study  results  will  be  used.  Study  results  are  more  likely  to  be  used  if: 
the.  study  management  principals  are  the  same  from  inception  to  imple- 
mentation; if  the  results  do  not  require  acquisition  of  large  amounts  of 
new  data  in  order  to  be  effective;  if  appropriate  authorities  take  approval 
actions  expeditiously;  and  if  the  results  are  delivered  on  time. 

11.  Benefits.  The  most  difficult  aspect  of  this  study  was  deter- 
mining the  benefits  which  had  accrued  as  a result  of  TASP  achievement. 

ESG's  analysis  addressed  tangible  and  intangible  benefits  resulting  from 
the  fact  that  a study  was  done.  Direct  benefits  as  well  as  indirect  and 
enduring  bonus  benefits  were  considered.  Figure  8 summarizes  ESG's 
findings . 
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Figure  8 


a.  ESG  found  that  those  studies  with  calculable  cost  effec- 

tiveness are  about  evenly  split  between  functional  and  Army-wide  impact 
and  that  75  percent  in  this  group  were  conducted  in-house.  The  cost- 
effectiveness  calculations  depend  on  several  essential  assumptions  which 
in  effect  determine  the  results.  These  assumptions  are  required  to 
answer  the  questions:  How  do  we  apportion  benefit/effectiveness  to  a 

study  distinct  from  other  decision  inputs?  And,  what  would  have  been  the 
alternative  chosen  if  the  study  had  not  been  done? 

b.  Benefits  realized  at  the  functional/local  level  were  found 
easiest  to  identify.  Those  most  difficult  to  determine  had  to  do  with 
long-range  planning,  strategy,  and  concepts. 


c.  When  a sponsor  decides  to  undertake  a study — whether  it  be 


in-house  or  contract,  they  generally  do  not  consciously  consider  expected 
benefit/payoff  in  a capital  budgeting  sense.  This  is  where  position/ 
authority  of  the  study  proponent  carries  the  most  weight.  It  becomes 
a question  of  "Who  said  we  need  this  study?"  If  the  person  in  the  position 
of  authority  departs  during  tht.  course  of  the  study,  there  is  a high  risk 
of  study  failure  and  most  likely  no  benefit. 

d.  ESG  also  found  that  reasonable  people  will  disagree  on  the 
extent  or  nature  of  the  benefit  from  any  particular  study.  Much  of  the 
benefit  depends  on  one's  perspective,  familiarity  with  the  problem,  and 
how  the  sc lution/decision  fits  in  with  your  own  execution  of  mission  and 


functions. 

e.  The  most  significant  indirect  benefit  is  reflected  in  the 
Army's  in-house  study  capability.  Available  in-house  study  resources  are 
judged  to  be  as  good  or  better  than  contractor  and  other  outside  resources. 
It  is  doubtful  that  the  current  level  of  expe"tise  would  have  developed 
without  learning  and  experience  from  past  study  efforts. 

f.  Conclusions: 

(1)  Cost-effectiveness  calculations  of  studies  are  not 
precise  and  may  even  be  meaningless  in  view  of  the  assumptions  required 
to  perform  such  calculations. 

(2)  There  appears  to  be  no  doubt  that  the  TASS  produces 
significant  benefits.  Definition  of  these  benefits  requires  an  appreci- 
ation of  the  problems  from  the  user's  perspective. 
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(3)  A priori  estimation  of  expected  study  benefits  is  not 
evident  in  the  study  justification  process.  Benefit  considerations  may 
be  implied,  however,  in  statements  asserting  the  importance  of  the 
expected  results  to  solving  a problem  or  getting  the  job  done. 

12.  Study  System  Data.  ESG  relied  largely  on  the  Army  studies 
portion  of  the  Defense  Documentation  Center  (DDC)  data  bank  for  study 
information  pertaining  to  FYs  74  and  75.  This  data  bank  is  particularly 
important  since  all  Army  studies  undertaken  are  required  to  be  reported. 

And,  after  completion  the  sponsor  enters  a results  and  use  evaluation. 

Among  other  things,  the  data  bank  should  be  useful  to  the  study  system 
managers  at  all  levels,  should  provide  visibility  to  completed  and  on- 
going efforts,  and  should  be  used  by  sponsors  of  proposed  studies  to 
preclude  duplication  of  effort.  In  the  course  of  this  study,  ESG  did 
evaluate  the  use  and  usefulness  of  the  DDC  data  bank. 

a.  The  most  frequent  user  of  the  data  bank  was  found  to  be  HQ 
DA  Study  Management  Office.  Use  by  that  office  exceeded  that  of  all 
other  users  combined. 

b.  On  15  December  1975,  the  ESG  study  team  obtained  a listing 
of  all  studies  reported  as  completed  or  terminated  during  FY  74  and  FY  75. 
That  listing  of  544  studies  was  found  to  contain  many  errors  and  duplicate 
entries.  Of  the  544  reported,  113  were  duplicate  entries.  The  study  team's 
last  use  of  the  data  bank  was  on  23  April  1976.  At  that  time,  479  studies 
were  listed  and  only  17  were  duplicate  entries — a large  improvement. 
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c.  Data  discipline  was  found  to  be  poor  despite  rather  detailed 
instructions  for  completing  the  DD  1498  forms  used  to  enter  the  study  . 
information.  Both  the  administrative  information  (i.e.,  resource  levels, 
categories,  sponsor,  study  agency)  and  study  substance  information  (purpose, 
objectives,  results,  use,  etc.)  are  often  ambiguous.  We  found  that  the 
information  reported  is  much  too  general  to  be  useful  as  a basis  for 
rigorous  evaluation  of  the  results,  uses,  and  benefits  of  completed  studies. 

d.  Although  ESG  did  not  find  obvious  duplication  of  study 
efforts,  it  is  not  clear  that  significant  credit  should  go  to  the  data 
bank.  It  was  found  that  coordination  of  the  individual  draft  study 
programs  and  the  review/ consolidation  at  HQ  DA  does  more  to  preclude 
duplication.  Groups  such  as  the  Logistics  Studies  Steering  Group  and 
the  recently  established  Strategic  Studies  Advisory  Group  also  play  an 
important  role  in  their  respective  areas. 

e.  Lack  of  knowledge  about  how  to  access  the  data  was  found  to 
discourage  use.  There  is  no  automatic  dissemination  of  completed  study 
information.  Thus,  prospective  users  must  exercise  their  initiative  if 
the  data  bank  is  of  value  to  them.— ^ 

f.  Completion  of  the  questionnaires  required  the  recall  of 
large  amounts  of  information  by  the  sponsoring  organizations  and  study 
agencies.  ESG  found  that,  for  the  majority  of  studies,  implementation 

6 J Twice  annually.  Director  of  Management,  OCSA  provides  each  DA 
Staff  and  MACOM  a listing  of  its  current  studies  cataloged  in  the  DDC. 
However,  each  DA  Staff  and  MACOM  does  not  automatically  get  any  listing 
of  what  others  have  cataloged. 
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and  use  information  depended  on  a human  memory  bank  rather  than  a for- 
mally established  data  file  (automated  or  not).  Failure  to  complete  a 
questionnaire  was  almost  always  due  to  the  nonavailability  of  someone 
familiar  with  the  study. 

g.  Conclusions. 

(1)  Quality  of  the  study  information  in  the  data  base  is 
being  improved  and  can  be  improved  further. 

(2)  Existing  data  are  not  widely  used  for  the  purposes 
intended  except  at  HQ  DA  level. 

(3)  Data  discipline  needs  to  be  enforced  in  the  preparation 
of  DD  Form  1498. 

(4)  Nature  and  content  of  automatic  (top-down)  dissemi- 
nation should  be  reexamined. 

(5)  Steps  must  be  taken  to  more  carefully  record  use, 
implementation,  and  benefit  information  as  part  of  the  data  file  for 
each  study. 

III.  DISCUSSION  AND  RECOMMENDATIONS 

13.  Study  Program  Focus.  There  is  no  easy  way  to  determine  the 
most  important  studies  in  relation  to  critical  Army  issues.  The  primary 
reason  for  this  is  that  critical  Array  issues  are  not  identified  in 
specific  terms.  The  SPG  is  not  sufficiently  discriminatory  to  be  an 
effective  focal  force.  The  DA  Staff/MACOM  parts  of  the  TASP  are 


separated  physically  and  functionally.  Each  part  is  successful  and  in 
focus  relative  to  its  own  mission  and  function. 

a.  Deference  to  requests  for  studies  from  higher  authority  has 
been  effective  in  meeting  nonprogramed  requirements-.  Implicit  in  this 
reaction  is  the  assumption  that  these  requests  are  critical  issues.  The 
disruption  which  results  has  an  overall  negative  effect  on  the  execution 
of  the  study  program.  The  Army's  acknowledged  study  capacity  has  very 
little  slack.  Even  so,  in-house  capability  is  more  responsive  than 
contract. 

b.  The  distinction  between  staff  actions  and  studies  is  not 
clear.  It  is  even  debatable  that  the  distinction  should  be  made,  and  the 
ESG  study  team  tends  to  oppose  continued  distinction.  The  issue,  problem, 
or  decision  involved  is  the  important  point.  Whether  "a  study"  is  the 
best  way  to  address  it  is  a management  decision  which  must  be  made  by 
executives  and  commanders  responsible  for  solving  the  problem.  We  have 
assumed  that  programing  decisions  by  the  executives  and  commanders  imply 
the  importance  of  the  issue  and  are  sufficient  justifications  for  under- 
taking a study  if  resources  are  available. 

c.  Resources  are  never  unconstrained,  and  someone  must  pick  and 
choose.  If  the  Army  wants  to  ensure  that  its  critical  issues  are  studied, 
these  issues  must  be  identified  and  resources  programed  accordingly.  The 
identification  should  be  made  by  top  management. 
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d.  Recommendations. 


(1)  Do  not  now  radically  change  the  manner  in  which  individ- 
ual agency /command  study  programs  are  developed. 

(2)  Top  management  should  make  early  identification  of  spe- 
cific key  issues  and  problems  toward  which  portions  of  the  study  capability 
can  be  planned  and  programed.  The  SPG  should  be  revamped  to  reflect  key 
issues  and  to  distinguish  between  important  and  less  important  priority 
areas . 

14.  Resource  levels.  Cost  and  manpower  data  acquired  for  the  studies 
evaluated  were  cumulative,  multi-FY  information.  Further,  ESG  was  only 
concerned  with  completed  and  terminated  studies.  For  these  reasons,  the 
question  of  overall  resource  adequacy  could  not  be  addressed.  The  data 
available  gave  no  indication  that  major  resource  imbalances  existed.  The 
analysis  and  synthesis  of  this  study's  early  findings  permit  recommending  a 
strategy  for  absorbing  resource  cutbacks  at  least  cost  to  program  effec- 
tiveness . 

a.  Due  to  the  way  in-house  studies  and  study  agencies  are  bud- 
geted, it  is  not  clear  how  selective  cutbacks  would  be  manifested.  All 
types  of  studies  do  not  require  the  same  amount  of  study  capability.  As 
another  variable,  many  of  the  technical  and  functional  areas  have  spe- 
cialized expertise.  There  is  risk  that  an  entire  area  of  expertise  could 
be  inadvertently  eliminated  by  across-the-board  cuts. 

b.  Nevertheless,  the  following  strategy  is  recommended: 
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(1)  Only  undertake  large-scale  model  and  simulation  efforts 
that  are  essential  and  for  which  acceptable  data  are  certain  to  be  avail- 
able. This  applies  to  both  in-house  and  contract  efforts.  (This  must  be 
done  cautiously  because  it  could  affect  the  productivity  of  the  study 
doers.  These  type  studies  result  in  tools  which  permit  analysts  to  do 
more  and  different  things  more  often  and,  hopefully,  better.) 

(2)  Only  undertake  widespread  methods  and  standardization 
efforts  that  address  critical  issues.  They  are  expensive,  time  consuming, 
and  tend  to  be  overtaken  by  events.  Applies  to  in-house  and  contract 
efforts. 

(3)  Curtail  contract  study  efforts. 

(4)  As  a last  resort,  absorb  cuts  in  in-house  capability. 

(5)  In  all  cases,  resource  levels  must  be  keyed  to  actual 
identified  need  rather  than  level  of  effort. 

15.  Improvements  in  Study  Planning  and  Programing. 

a.  In  addition  to  the  issues  of  program  focus  and  SPG,  there 
are  study  characteristics  which  help  to  ensure  success.  They  are  stated 
here  as  criteria  for  use  when  evaluating  study  proposals.  The  criteria 
were  developed  from  analysis  of  all  the  sampled  studies  which  were  classed 
as  meeting  all  their  objectives  and  were  used. 

b.  Recommendations.  When  evaluating  study  proposals,  the 
following  conditions/criteria  should  be  met: 

(1)  Clearly  defined  problem  and  purpose. 
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(2)  Identified  need  in  terms  of  problem  and  expected  results 

and  uses. 

(3)  Manageable  number  of  objectives;  consistent  scope  and 
resource  estimate. 

(4)  Statement  of  expected  results,  anticipated  uses,  and 
explicitly  identified  user. 

(5)  Assurance  of  high-level  sponsor  interest. 

(6)  SAG,  steering  group,  or  similar  study  management  specified. 

16.  Results,  Uses,  and  Benefits.  Two  fundamental  problems  seem  to 

plague  this  area  even  where  focus,  resource  levels,  and  planning/programing 
have  been  proper:  discor.tinuities  in  study  management  and  paucity  of 

documentated  evidence  of  uses  and  benefits.  The  former  is  by  far  the  most 
important. 

a.  Results,  uses,  and  benefits  of  studies  often  depend  on  a 
specific  individual.  This  is  more  apparent  on  the  sponsor  management 
side  than  on  the  study  agency  side.  Changes  in  action  officers,  executives, 
and  commanders  can  cause  partial  use  or  nonuse  of  results  of  completed 
studies;  in  other  cases,  studies  that  are  ongoing  can  flounder.  New 
management  often  has  different  perceptions  of  the  problems  and  the  manner 
in  which  they  will  be  studied.  In  addition  to  management  continuity, 
there  must  be  management  interest  particularly  at  high  levels  in  the 
sponsoring  organization.  If  executives  and  commanders  are  personally 
interested  in  a study,  its  completion  and  use  are  almost  certain. 
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b.  Study  achievements , uses,  and  benefits  certainly  are  not  as 
veil  known  as  the  Army  would  wish.  Implementation  actions  and  effects 
are  not  uniformly  documented,  if  at  all.  As  previously  noted,  results  and 
use  evaluations  reported  to  the  data  bank  are  of  doubtful  utility  for  a 
variety  of  reasons.  If  study  results,  uses,  and  benefits  are  to  be  assessed 
periodically  for  program  evaluation  purposes,  a more  effective  method  of 
acquiring  the  information  needs  to  be  instituted. 

c.  Recommendations . 

(1)  The  problem  of  action  officer  and  SAG  turbulence  should 
be  addressed  and  resolved  by  top  management. 

(2)  Executives  and  commanders  need  to  become  more  involved 
early — when  the  need  for  a study  is  established.  And,  they  need  to 
continue  expressing  interest  throughout  execution  and  implementation.  It 
may  be  advisable  to  establish  a threshold  (say  3 PMY  or  $150K)  above  which 
the  executive/commander  must  personally  approve  the  study. 

(3)  Current  requirements  to  report  results,  uses,  and  benefits 
data  need  to  be  examined  to  ensure  that  data  are  adequate  for  program 
evaluation  purposes.  It  should  be  mandatory  that  all  such  data  be 
recorded  before  all  parties  to  the  study  execution  and  implementation  have 
departed.  One  important  consideration  which  should  be  built  into  the 
implementation  plan  is  a specific  provision  for  gathering  information 
necessary  to  conduct  a comprehensive  and  valid  appraisal  of  results  and 


17.  Study  Information. 
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a.  The  inadequacies  of  the  formal  study  information  system  (e.g., 
study  portion  of  DDC)  are  compounded  by  the  study  community's  general  lack 
of  knowledge  about  the  state  of  the  study  program  at  a given  time.  Use 

of  study  results  depends  on  knowing  when  results  are  available  and  what 
they  are.  Although  results  of  one  study  may  not  apply  directly  to  other 
problems,  there  are  lessons  learned,  novel  or  new  methodologies,  data  and 
data  sources,  and  other  features  that  can  be  exploited  by  the  study 
community  at  large.  Interest  expressed  by  others  in  one's  achievements 
also  should  result  in  some  feeling  of  satisfaction  and  serve  to  encourage 
quality  performance. 

b.  Recommendation. 

(1)  Publish,  at  least  quarterly,  a list  of  all  studies 
completed  or  terminated  during  that  period.  That  list  should  Include 
the  name  of  study  agency  and  point  of  contact  and  should  be  distributed 
to  all  study  coordinators. 

(2)  Institute  a study  newsletter  to  get  the  word  out  to 
commanders,  executives,  study  agencies,  and  analysts  regarding  what  is 
happening  in  the  study  community.  The  newsletter  should  include  signif- 
icant accomplishments,  continuing/current  technical  problems,  and 
developing  issues  which  may  require  study.  The  newsletter  should  be 
distributed  to  all  through  executive  channels. 
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18.  General  Observation. 
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a.  AR  5-5  (and  DOD  Directive  5010.22)  was  being  revised  while 
this  effort  was  underway.  The  information  available  to  ESG  indicated 
that  the  revised  AR  will  contain  the  words  necessary  to  achieving  an 
effective  TASS.  ESG  does  not  believe,  however,  that  this  AR  solves  the 
issue  previously  raised  concerning  SPG.  The  system  will  remain  only  words 
unless  it  is  enforced. 

b.  Recommendation.  The  Study  Management  Office  should  closely 
monitor  and  enforce  compliance  with  AR  5-5  (and  its  companion  DA  pamphlet) . 
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ANNEX  A 


DATA  COLLECTION— QUESTIONNAIRES— INTERPRETATION 
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1.  Purpose  and  Scope.  This  annex  presents  a somewhat  detailed 
discussion  of  the  processes  involved  in  acquiring  and  interpreting  the 
study  data.  It  includes  discussions  of  sample  selection,  data  collec- 
tion, and  a variety  of  tabulations.  Appendixes  A-l  and  A-2,  respec- 
tively, contain  Type  I and  Type  II  Questionnaires  annotated  to  show 
consolidated  responses. 

2.  Sample  Selection. 

a.  Sponsor  guidance  specified  that  studies  completed  or  termi- 
nated during  FY  74  and  FY  75  would  be  candidates  for  evaluation.  In 
order  to  determine  which  studies  qualified  for  evaluation,  ESG  used  the 
terminal  in  the  Army  Library  to  query  the  Army  studies  portion  of  the 
DDC  data  bank.  (Army  Library  personnel  were  most  helpful  in  this  and 
all  subsequent  uses  of  the  terminal  by  the  ESG  study  team.)  The 
initial  listing  contained  544  studies.  Examination  of  the  printout 
disclosed  many  duplications  and  "carrier  studies. "IV  In  fact,  113 
were  deleted  from  the  listing  (61  carrier  and  52  duplicate  studies) . 
Examination  of  available  agency/MACOM  study  programs  for  FYs  75,  76, 
and  7T  disclosed  31  completed  studies  which  were  not  in  the  listing. 
(Agency/MACOM  annual  study  programs  identify  studies  completed  in  the 
previous  fiscal  year.)  With  the  above  adjustments  to  the  DDC  listing, 

1 7 Carrier  studies  can  best  be  described  as  integrating  devices 
which  relate  a group  of  separate  studies.  A carrier  study  is  useful  from 
a management  perspective  but  is  not  a study.  The  term  "umbrella"  study 
is  also  used. 
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there  were  462  studies  as  possible  candidates  for  inclusion  in  the 

i 2/ 

sample.— 

b.  The  462  studies  were  distributed  by  TASP  category  and  means 
of  accomplishment  as  shown  in  Figure  A-l.  Original  plans  called  for 
each  study  in  the  sample  to  be  evaluated  by  on-site  interviews.  Con- 

V 

sidering  the  time  and  resources  available,  ESG  decided  that  no  more  than 
80  studies  could  be  evaluated.  These  80  studies  were  drawn  through  a 
stratified  random  selection  which  retained  the  general  category  and  in- 
house  vs  contract  proportions  of  the  study  population.  An  additional 
requirement  specified  at  least  two  studies  in  each  cell.  Figure  A-2 
shows  the  randomly  drawn  sample. 

c.  Thirty-one  high-cost  studies  not  included  in  the  randomly 
drawn  sample  were  identified.  High  cost  was  defined  as  more  than  300,000 
dollars  for  contract  or  6 PMY  for  in-house  studies.  These  were  added  to 
the  group  to  be  evaluated.  Agencies  and  MACOMs  were  also  asked  to  nomi- 
nate up  to  three  successful  and  three  not  so  successful  studies  for 
evaluation.  Thirty-four  studies  not  in  the  random  draw  or  high-cost 

3/ 

categories  were  nominated.  The  evaluation  sample  totaled  145  studies.— 

Annex  B contains  a listing  identifying  each  study  by  title. 

2/  A complete  dump  of  the  DD  Forms  1498  on  23  Apr  76  showed  that 
the  data  bank  was  being  policed.  Of  the  479  studies  listed,  only  30 
were  duplicates  o::  carrier  studies  and  14  of  the  31  previously  not 
reported  were  included. 

3 J Hereafter  the  use  of  the  word  "sample"  refers  to  all  145  studies 
unless  qualified  with  "random"  or  "high  cost." 
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3.  Data  Collection.  Mailed  questionnaires  were  the  primary  instru- 


ment for  collecting  data  on  the  145  studies.  Two  types  were  developed: 
Type  I for  the  individual  study  and  Type  II  for  the  study  process  in 
general  (see  Appendixes  A-l  and  A-2) . Both  were  to  be  executed  by  knowl- 
edgeable people  within  agencies  and  MACOMs.  In  addition  to  the  ques- 
tionnaires, major  use  was  made  of  the  data  bank  and  facsimile  printouts 
of  each  study's  DD  Form  1498.  Available/locatable  copies  of  study  reports 
and  related  documents  were  used  also. 

a.  Type  I Questionnaires  were  mailed  to  sponsors  and  known 
users  (other  than  sponsors)  of  the  studies  in  the  sample.  In  all,  178 
questionnaires  were  sent,  33  studies  being  covered  by  two  questionnaires. 
Addressees  were  allowed  4 weeks  to  complete  the  questionnaire.  In  the 
meantime,  the  study  team  began  independently  evaluating  a group  of 
studies  from  the  sample  in  order  to  assess  the  adequacy  of  the  replies 
and  to  verify  questionnaire  responses.  The  study  team  completed  34 
study  evaluations.  ESG's  evaluation  consisted  of  examining  study  docu- 
ments (reports,  taskers,  implementation  instruments,  and  related 
materials)  and  contacting,  primarily  by  phone,  their  users  or  preparers. 

b.  Type  I Questionnaire  returns  totaled  161  or  90  percent  of 
those  mailed.  They  pertained  to  134  out  of  the  145  studies  in  the  sample 
(92  percent).  ESG .considered  the  response  excellent.  Before  tabulat- 
ing the  data,  ESG  compared  its  independent  evaluations  with  the  corre- 
sponding questionnaires  and  found  reasonably  close  agreement.  Separate 


sponsor  and  user  assessments  also  tended  to  agree.  Questionnaire  data 


were  augmented  with  data  acquired  from  study  reports,  files,  and  the 
DD  1498s.  Figure  A-3  summarizes  the  individual  study  data  acquisition 
effort. 

INDIVIDUAL  STUDY  DATA  ACQUISITION 


No.  of 

Action Studies 

178  Questionnaires  Mailed  145 

161  Questionnaires  Returned  134 

Independent  Evaluation  34 

Examination  of  Reports /Files  91 

Examination  of  DD  1498  141 


Figure  A-3 

c.  Type  II  Questionnaires  were  mailed  to  agencies  and  MACOMs 
that  develop  annual  study  programs.  The  questionnaire  sought  to  obtain 
information  about  program  development  procedures,  past  program  experi- 
ences, cost  effectiveness  and  benefits,  and  suggested  improvements. 
Sixteen  questionnaires  were  mailed,  and  14  were  returned.  Appendix  A-2 
is  a consolidated  version  of  the  questionnaire. 

4.  Resource  Expenditures.  The  462  studies  completed  in  FY  74  and 
FY  75  required  approximately  1,363  man-years  of  in-house  effort  and 
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39  million  dollars  of  contract  effort  as  shown  in  Figure  A-4.  Converting 
the  in-house  effort  at  the  rate  of  50,000  dollars  per  man-year  results  in 
an  overall  cost  approaching  110  million  dollars.  It  must  be  remembered 
that  these  are  cumulative  costs  covering,  in  most  cases,  more  than  one 
fiscal  year.  They  do  not  reflect  any  single  year  costs  of  TASP.  The 
figures  include  proportions  of  the  resources  spent  in-house  and  by  con- 
tract . 

a.  Figure  A-5  shows  similar  cost  data  for  the  sample.  In 
assembling  and  checking  the  data,  it  became  apparent  that  contract  cost 
data  are  much  more  accurate  than  in-house  PMY  data.  In-house  PMY  costs 
generally  originate  as  and  remain  estimates.  Comparison  of  DD  1498 
cost  data  with  questionnaire  responses  showed  almost  complete  agree- 
ment in  the  contract  costs.  In-house  cost  data  differed  by  more  than 
1 PMY  in  15  percent  of  the  cases  and  by  .5  PMY  in  47  percent  of  the 
cases.  In  79  percent  of  these  instances,  questionnaire  responses  were 
higher  than  the  DD  1498.  Sixty-four  percent  of  the  "unknown"  responses 
to  the  cost  question  (17  percent  of  all  responses)  pertained  to  in-house 
data.  This  suggests  that  the  closely  regulated  and  fairly  uniform 
contract  study  bookkeeping  may  be  the  reason  for  data  consistency. 
Sponsoring  elements  apparently  do  not  routinely  acquire  and/or  retain 
comparable  cost  data  for  in-house  studies. 
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Figure  A-4 


Figure  A-5 
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b.  Because  there  was  disparity  between  data  bank  and  question- 
naires with  regard  to  cost  data,  ESG  elected  to  use  the  DD  1498  figures 
for  the  sake  of  consistency.  Figures  A-4  and  A-5  are  based  on  DD  1498 
data. 

c.  Similarly,  the  category  assignment  of  a particular  study  is 
based  on  the  DD  1498.  There  appears  to  be  a great  deal  of  flexibility 
in  designating  the  category  to  which  a study  will  be  assigned.  One  out 
of  three  responses  assigned  the  study  to  a category  other  than  that 
reported  to  the  data  bank.  Examination  of  these  studies'  problems, 
purposes,  and  objectives  disclosed  that,  indeed,  it  was  quite  likely 
that  different  people  could  legitimately  specify  different  categories 
depending  on  their  viewpoint  or  area  of  emphasis.  Because  the  cate- 
gories are  not  mutually  exclusive,  it  did  not  appear  useful  or  enlight- 
ening to  pursue  the  question  of  category  "level-of-ef fort ."  If  the 
categories  are  to  be  used  as  a resource  management  device,  there  must 

be  a strict  method  of  assigning  a study  to  a category. 

d.  Using  the  DD  1498  data  for  all  462  studies,  the  costs  of 
the  "average"  study  were  calculated.  Figure  A-6  shows  that  there  is 
not  a great  deal  of  difference  between  the  average  in-house  and  contract 
costs . 

5.  Identifying  and  Planning  for  Studies.  The  investment  in  studies 
is  by  no  means  insignificant.  Who  identifies  the  need  for  this  commit- 
ment of  resources?  Throughout  TASS's  history,  top  management  has  been 
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concerned  that  Army  resources  were  being  used  in  some  cases  for  studies 
that  aade  little  sense  in  terns  of  their  relation  to  genuine  Army  prob- 
lems and  missions.  Concern  has  also  been  expressed  that  the  real  criti- 
cal Issues  may  not  be  adequately  studied.  Part  of  the  questionnaire 
addressed  problem  identification  and  planning  for  study  efforts. 


THE  "AVERAGE"  STUDY  COST 
(FY  74  and  FY  75) 


In-house 

Contract 

Combined 

Number 

Category 

Man- 

years 

Dollars 
(K)— ' 

Dollars 

00 

Average 

Cost 

of 

Studies 

1 

7.48 

374.0 

328.5 

355.9 

33 

2 

3.55 

177.3 

205.3 

187.7 

78 

3 

5.86 

293.1 

303.4 

297.2 

107 

4 

4.72 

236.2 

292.2 

248.7 

112 

5 

2.44 

122.0 

177.8 

143.1 

106 

6 

4.87 

243.4 

179.7 

226.3 

26 

Overall 

4.47 

223.5 

248.0 

231.8 

462 

a/  One  ma-.-year  = $5QK. 


Figure  A-6 

a.  Origins  of  study  requirements.  ESG  thought  it  important  to 
know  the  extent  to  which  study  sponsors  are  told  (or  asked)  to  undertake 
particular  studies.  Figure  A-7  shows  the  findings  based  on  the  Type  I 
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Questionnaire,  question  12.  Although  the  proportions  pertain  to  the 
sample  and  may  not  be  entirely  transferable  to  studies  in  general,  it 
seems  safe  to  say  that  a significant  proportion  of  an  agency-  or  MACOM- 
sponsored  study  program  is  devoted  to  problems  or  issues  which  origi- 
nate from  above.  This  result  was  compared  with  the  responses  to  questions 
13  and  17  of  the  Type  II  Questionnaire.  Question  13  asked  how  priorities 
are  assigned,  and  17  asked  for  the  most  frequent  cause  of  deferring  or 
terminating  a study.  Responses  to  these  two  questions  can  be  summarized 
simply  as  "higher  headquarters'  requests."  Highest  priority  is  given  to 
requests  from  above  and  causes  deferral  and/or  termination  of  programed 
studies . 


ORIGINS  OF  STUDY  REQUIREMENTS 


DA 

Staff 

MACOM 

Total 

TASP 

Originator 

% 

Number 

% 

Number 

% 

Number 

DOD/JCS 

10.7 

8 

4.6 

3 

7.8 

11 

CSA/SA 

16.0 

12 

13.6 

9 

14.9 

21 

Staff  Agency 

66.6 

50 

27.3 

18 

48.3 

68 

MACOM 

1.3 

1 

48.5 

32 

23.4 

33 

Other 

2.7 

2 

3.0 

2 

2.8 

4 

Unsolicited 

Proposal 

2.7 

_2 

3.0 

_2 

2.8 

4 

Total 

100.0 

75 

100.0 

66 

100.0 

141 

Figure  A-7 
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b.  Agency/MACOM  study  program  development.  Questions  7,9,  and 
10  of  the  Type  II  Questionnaire  addressed  program  development.  The 
responses  are  summarized  as  follows. 

(1)  The  requirements  of  AR  5-5  are  followed.  Most  agencies 
and  MACOMs  have  augmented  the  procedures  of  the  AR  with  local  supplemen- 
tary instructions  and  procedures. 

(2)  SPG  has  marginal  to  important  influence  on  program 
focus  but  is  not  a decisive  factor. 

(3)  Agency/MACOM  percepcion  of  its  mission  and  its  function 
responsibilities  is  the  most  important  factor  in  determining  study  pro- 
gram content. 

c.  Interpretation.  Reflecting  on  the  responses  noted  above  and 
the  study  program  development  process  specified  in  TASS,  the  following 
observations  were  made. 

(1)  This  deferrence  to  higher  headquarters  study  require- 
ments implies  that  the  problem  or  issue  from  that  source  is  of  higher 
priority.  However,  the  responses  tend  to  show  that  priority  is  related 
more  to  who  has  the  problem  than  to  what  is  the  problem.  ESG  is  not  in 
a position  to  say  that  this  is  good  or  bad,  but  suggests  later  that  the 
authority-problem  relationship  can  be  exploited  to  ensure  successful 
completion  and  use  of  studies. 

(2)  SPG  does  not  appear  to  serve  its  intended  purposes.  ESG 
thinks  it  is  not  sufficiently  discriminatory  to  be  a useful  yardstick 


for  evaluating  the  appropriateness  of  agency /MACOM  study  programs. 

Because  the  SPG  is  developed  from  the  bottom  up,  it  essentially  reflects 
local  mission  and  function  issues — making  it  easy,  should  the  need  arise, 
to  justify  almost  any  study  based  on  identity  with  an  SPG  priority  area. 
Within  and  among  the  priority  areas,  there  are  no  orders  of  importance 
(e.g. , priority  of  the  priority  areas).  In  short,  it  is  still  difficult 
to  translate  the  SPG  into  a comprehensible  set  of  specific  critical 
decisions  or  problems  requiring  study.  It  is  not  surprising  that  most 
people  queried  consider  the  SPG  marginally  important  in  providing  focus 
to  their  annual  study  programs. 

(3)  From  an  agency/MACOM  perspective,  the  study  programs 
seem  properly  oriented  toward  solving  mission  and  function  problems. 

For  their  own  generated  programs , agencies  and  MACOMs  have  implemented 
systems  for  establishing  priorities  based  on  the  importance  of  the  issue, 
resources  available,  and  the  guidance  of  the  executive  or  commander. 
Several  of  these  systems  are  formally  established  and  documented. 

6.  Study  Results.  One  goal  of  this  undertaking  was  to  determine 
if  studies  achieved  what  they  set  out  to  do.  ESG  also  wanted  to  deter- 
mine characteristics  associated  with  successful  and  unsuccessful  studies. 
Respondents  were  asked  for  their  opinions  about  study  execution  and 
results . 
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a.  As  shown  in  Figure  A-8,  45  percent  of  the  sample  (64  studies) 
failed  to  achieve  one  or  more  of  their  objectives.  Respondents  were 
asked  to  indicate  why  objectives  were  not  met  (question  29,  Type  I Ques- 
tionnaire). In  general,  there  were  multiple  reasons  for  not  meeting  a 
particular  objective.  Following  are  the  most  frequently  mentioned  causes. 

(1)  Time  or  resources  did  not  permit  covering  all  objec- 
tives. Too  many  objectives. 

(2)  Objective  was  to  develop  a method,  system,  or  model 
for  some  activity  which  turned  out  to  be  ill-defined  or  nonstable  (i.e., 
undergoing  rather  rapid  change). 

(3)  The  DA  Staff  and  CONUS  reorganization  primarily  affected 
studies  which  had  been  initiated  or  sponsored  by  elements  which  did  not 
survive  the  reorganization.  In  some  cases,  the  problem  (o^  the  need) 
apparently  disappeared  as  a result  of  the  reorganization. 

(4)  There  were  changes  within  the  sponsoring  organization 
(action  officers)  and  study  agency.  Such  changes  affect  the  perceived 
need  for  or  focus  of  the  study. 

(5)  As  the  study  progresses,  the  problem  proves  less  tract- 
able than  originally  thought  or  the  available  analytic  tools  are  found 
unsuitable . 
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OBJECTIVES  MET 


Ob  ice Lives 


Percent  Number 


All  Met 

55 

77 

More  Than 

Half 

Met 

30 

42 

More  Than 

Half 

Not  Met 

9 

13 

None  Met 

6 

9 

Total 

100 

141- 

a/  Returned  were  134  questionnaires.  DD 
1498s  were  the  source  of  data  for  seven  studies. 
Four  studies  were  not  included  because  of  inade- 
quate information. 


Figure  A-8 

b.  In  a later  question  (question  39,  lype  I Questionnaire), 
respondents  were  asked  to  indicate  what  should  have  been  done  differently 
to  improve  the  success  of  their  study.  ESG  wanted  to  ascertain,  among 
other  things,  their  opinions  regarding  the  weaknesses  in  the  execution 
of  the  particular  study.  Not  all  people  responded  to  this  question.  Of 
the  161  questionnaires  returned,  96  pertaining  to  89  studies  answered 
question  39.  The  responses,  however,  are  perceptive.  The  narrative 
respom.es  are  summarized  as  follows. 

(1)  Improvements  related  to  planning  and  programing  in  the 
broad  sense  were  most  frequently  mentioned.  A few  examples  of  such 
statements  are:  "should  have  been  done  earlier;"  "should  have  allowed 
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more  time  and/or  resources;"  "should  not  have  changed  their  minds;"  and 
so  on,  all  related,  we  believe,  to  planning  and  programing. 

(2)  The  second  most  frequently  mentioned  improvement  had  to 
do  with  stability  in  proponency  and/or  lack  of  interest  "where  it  counted. 
In  some  respects , the  lack  of  demand  or  motivation  to  produce  was  com- 
bined with  changes  in  study  direction  due  to  proponent  changes. 

(3)  Other  types  of  responses  did  not  fall  into  any  partic- 
ular pattern.  They  ranged  from  "would  not  do  it  at  all"  to  "no  change." 

7.  Use  of  Study  Results.  Questions  32,  33,  34,  and  37  of  the 
Type  I Questionnaire  and  question  11  of  the  Type  II  Questionnaire  solicit 
information  about  uses  and  users  of  study  results.  Figure  A-9  shows  how 
studies  were  used.  Since  studies  can  have  multiple  uses,  they  were 
classed  according  to  the  use  which  most  closely  matched  their  original 
purpose.  Classification  was  a study  team  judgment  based  on  questionnaire 
and  DD  1498  interpretation. 

a.  The  19  studies  that  were  not  used  included  three  for  which 
there  was  insufficient  information  to  say  with  any  confidence  that  they 
were  or  were  not  used.  ESG  thought  that  grouping  them  with  Not  Used 
would  be  the  safest  approach.  Nine  were  not  used  because  they  either 
achieved  no  objectives  or  were  terminated  before  completion.  Reasons 
for  nonuse  of  the  others  identified  by  the  respondents  were: 

(1)  Problem  passed  with  time  or  changed  significantly. 
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(2)  Study  a complete  failure  or  suffered  from  unconvincing 


methodology /logic. 


(3)  Departure  of  individuals  who  expressed  the  need. 


(4)  Study  achieved  its  objectives,  but  Lae  implementation 


has  not  been  approved. 


(5)  Insufficient/unaccep table  input  data;  process  has 


changed . 


USE  OF  STUDY  RESULTS 


Type  Use 


Percent 


Number 


Direct  Input  to  Decision/Problem 
Solution 


Input  to  Planning  Process 


Input  to  Another  Study  (Phase) 


Reference  Data/Planning  Factor 


Not  Used 


Total 


100.0 


Figure  A-9 


b.  The  last  cause  of  nonuse  is  probably  the  most  important 


because  large-scale  model/simulation , method  improvement , and  process 


standardization  type  studies  are  about  half  of  the  Not  Used  category. 


These  undertakings  are  usually  among  the  most  expensive  and  time  con- 


suming. The  sample  information  suggests  that  before  undertaking  such 


efforts,  one  should  ensure  that  the  system  or  process  is  tractable  and 
that  the  inputs  essential  to  the  product  do  exist  or  are  sure  to  be 
developed. 

c.  In  order  to  determine  the  necessity/criticality  of  the  study 
results  and  use,  ESG  asked  (question  AO,  Type  1 Questionnaire)  what  the 
impact  would  have  been  if  the  study  had  not  been  done.  The  response  to 
the  question  was  narrative  so  some  interpretation  was  required.  ESG 
defined  significant  impact  as  whenever  a study  was  widely  used  and/or 
the  results  were  essential  input  to  a decision  or  problem  solution. 

Figure  A-10  relates  the  impact  with  achievement  of  objectives.  The 
marginal  impact  category  included  studies  of  local  interest.  It  must  be 
noted,  however,  that  the  impact  at  the  local  level,  in  most  cases,  was 
significant.  The  sponsor/user  and  ESG  team  could  not  form  opinions  on 
the  impact  of  nine  studies.  Some  of  these  were  planning  factor  and 
reference  document  studies.  The  "no  impact”  response  for  the  studies 
which  achieved  no  objectives  is  self-explanatory  and  reasonable.  The 
other  no-impact  responses  were  somewhat  surprising.  The  respondents, 
however,  thought  there  would  have  been  no  impact  even  if  the  studies 
had  not  been  done.  Sue’’  studies  were  not  used  or  they  were  not  essen- 
tial to  a decision  or  problem  solution.  In  four  of  the  cases,  the 
respondents  Lelt  the  decision  had  already  been  made  and  the  study  was 
superfluous . 
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IMPACT  OF  STUDIES 


Objectives 


Impact 

Significant  Marginal  None  Unknown 


None  Met 


Most  Not  Met 


Most  Met 


All  Met 


Total 


Figure  A- 10 


8.  Benefits . Individually  and  overall,  the  benefits  from  studies 
are  extremely  difficult  to  specify  in  concrete  terms.  Questions  35  and 
36  of  the  Type  I Questionnaire  and  18,  19,  and  20  of  the  Type  II  Ques- 
tionnaire pertained,  respectively,  to  individual  study  and  overall  pro- 
gram benefits.  Respondents  generally  felt  that  a cost-effectiveness  or 
cost-benefit  approach  was  inappropriate.  While  all  felt  that  studies 
were  worthwhile,  there  was  little  in  the  way  of  measurable  value  asso- 
ciated with  the  feeling.  The  responses  were  broad  qualitative  statements 
in  about  85  percent  of  the  cases.  We  interpreted  "not  applicable" 
responses  to  mean  that  the  respondent  felt  that  the  measurement  of  bene- 
fits for  his  particular  study  was  inappropriate.  Because  benefits  were  not 
well  defined  in  the  response,  ESG  used  judgment  to  determine  and  cate- 
gorize benefits  for  each  study.  The  judgment  was  influenced  by  "who 


benefited,"  how  the  study  was  used  relative  to  decisionmaking/problem 


4 


solving,  and  the  nature  of  the  impact  of  using  the  study  results.  Figure 
A-ll  shows  the  findings  that  pertain  to  benefits. 


BENEFITS 

Type 

Fercent 

Number 

Calculable  Cost 

Effectiveness 

16 

22 

Significant 

52 

73 

Low 

21 

30 

None 

11 

16 

Total 

100 

141 

Figure  A-ll 


a.  Cost-effectiveness  figures  can  be  spurious.  In  order  to 
make  such  calculations,  one  must  assume  the  degree  of  influence  that  the 
study  had  in  decisionmaking/problem  solving.  (It  is  practically  impos- 
sible to  recreate  the  original  decisionmaking/problem-solving  eiviron- 
ment.)  Also,  in  the  absence  of  other  information,  one  must  assume  what 
decision/solution  would  have  been  selected  without  the  study.  In  effect, 
the  assumptions  determine  the  cost-effectiveness  rating — a very  unsatis- 
factory state  of  affairs.  Nevertheless,  16  percent  of  the  studies  were 
applicable  to  decisions /problems  which  resulted  in  quantifiable  (in 
dollars)  results.  These  studies  were  about  evenly  split  between  those 
having  functional  and  Army-wide  impact. 


A-21 


b.  Significant  benefit  is  a judgment.  A study  was  rated  as 
having  significant  benefit  if  it  was  implemented/used  as  intended, 
resulted  in  officially  recognized  improvements,  or  had  widespread  impact. 
Benefits  usually  cannot  be  specified  quantitatively,  and  the  actual 
measure  frequently  must  wait  the  test  of  time.  Overall,  significant 
benefit  suggests  very  attractive  returns  on  resource  investment. 

c.  Low-benefit  rating  is  also  judgmental.  These  studies  had 
limited  impact  (even  though  used),  had  obvious  indirect  value,  and/or  are 
not  currently  used  but  are  salvageable . These  are  considered  marginal 
returns  on  resource  investments . 

d.  No-benefit  rating  means  that,  based  on  questionnaire 
responses  and  ESG's  research,  the  study  efforts  were  of  no  identifiable 
direct  or  indirect  value. 

e.  Indirect  benefits  from  study  undertakings  are  probably  best 
demonstrated  by  the  capacity  and  capability  of  the  Army's  in-house  study 
resources.  The  response  to  question  21,  Type  II  Questionnaire,  indicates 
that  the  in-house  resources  are  equal  to  or  better  than  contract.  From 
an  overall  perspective,  none  of  the  data  comparisons  suggest  any  other 
assessment.  It  is  doubtful  that  current  levels  of  expertise  would  exist 
if  the  Army  had  not  learned  from  past  efforts.  In  addition  to  learning 
(both  from  success  and  failure),  other  indirect  benefits  include  reusable 
study  tools  (methods,  models,  systems)  and  many  unintended  spinoff  uses 
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of  study  results.  A spinoff  use  would  be  a study  now  used  as  a reference/ 
data  source  which  originally  was  used  in  a one-time  decision  or  problem 
solution. 

f.  Expected  benefits  do  not  seem  to  drive  decisions  to  commit 
resources  to  a study.  As  indicated  earlier,  the  position  and  authority 
of  the  element  requiring  the  study  seems  to  be  the  most  important  deter- 
minant of  whether  or  not  a study  will  be  undertaken.  More  than  one 
respondent  indicated  that  some  problems /decisions  will  be  addressed, 
whether  or  not  we  call  them  studies,  in  executing  their  mission  and 
function  responsibilities.  The  benefit  from  study  support  to  planning, 
readiness,  equipment,  organization,  and  so  on,  they  argue,  must  be  ulti- 
mately tested  on  the  field  of  battle. 

9.  Summary  Observations. 

a.  When  interpreting  the  previous  discussion  and  the  question- 
naires in  the  appendixes,  one  must  realize  that  the  data  are  based  on 
opinions.  ESG  believes  that  che  responses  are  sincere  and  based  on  the 
best  available  information.  Time  for  completing  the  questionnaires  and 
performing  the  sny thesis  and  analysis  was  limited.  ESG  did  not  try  to 
"audit"  any  study  or  judge  the  quality  of  the  study  process.  It  seems 
apparent,  however,  that  studies  which  are  used  widely  do  achieve  useful 
results. 

b.  There  are  certain  issues  that  were  not  fully  developed  in 
specific  terms.  They  are: 
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(1)  Do  studies  adequately  address  the  critical  Army  issues? 
Even  in  retrospect , it  is  not  clear  which  were  the  critical  issues  in 

FY  74  and  FY  75.  Therefore,  ESG  could  not  determine  whether  these  issues 
were  studied.  The  study  team  became  convinced  that  the  quality  and 
precision  of  guidance — the  critical  issue  forecast — are  very  important 
in  this  area. 

(2)  Resource  allocations,  in-house  vs  contract,  do  not 
necessarily  reflect  the  current  situation.  FCRC  affiliations  and  level- 
of-effort  arrangements  existed  when  most  of  the  contract  and  contract/ 
in-house  studies  completed  in  FY  74  and  FY  75  were  undertaken.  None  of 
the  data  supports  a strong  position  regarding  how  resources  should  be 
split.  The  findings  indicate  that  any  required  resource  adjustments 
should  be  made  on  a c.ase-by-case  basis  considering  the  issue,  need,  and 
most  timely  means  of  accomplishment.  A workable  priority  system  is  fun- 
damental to  such  a strategy. 

c.  The  accuracy  of  the  data  is  questionable  in  many  areas. 
Because  the  reporting  requirements  are  relatively  new  (FY  74) , some 
inconsistencies  are  to  be  expected.  But  if  study  management  wishes  to 
use  the  studies'  portion  of  the  DDC  data  bank  for  program  evaluation  and 
individual  study  evaluation,  the  data  and  reporting  discipline  needs  to 
be  improved.  Unless  the  data  are  recorded  in  a useful  form  as  events 
occur  (i.e. , completion,  implementation,  use,  benefits),  it  is  difficult 
to  recall  the  important  information.  People  move  frequently  and  the 
organizational  memory  is  poor  in  many  areas  of  study  information. 

LAST  PAGE  OF  ANNEX  A 
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APPENDIX  A-l 


QUESTIONNAIRE:  ARMY  STUDIES— RESULTS,  USES,  AND  BENEFITS 

TYPE  I 


*************************  EVALUATION  TEAM  USE  ONLY**************************** 


* 

* 

* 

* 

* 

Agency/Command : 

178  Mailed;  161  Returned 

* 

* 

* 

* 

Category:  TASP: 

(0-1) 

* 

* 

* 

* 

Sample: 

(0-2) 

* 

* 

* 

* 

Method : 

(0-3) 

* 

* 

* 

* 

Interview: 

* 

* 

* 

* 

* 

Analyst : 

* 

* 

****************************************************************************** 
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QUESTIONNAIRE:  ARMY  STUDIES— RESULTS , USES,  AND  BENEFITS 


TYPE  I 
INTRODUCTION 


The  purpose  of  this  questionnaire  is  to  aid  in  gathering  data  for 
use  in  evaluating  the  results,  uses,  and  benefits  of  Army  studies.  Use 
of  this  questionnaire  makes  it  possible  for  a larger  numbtr  of  studies 
to  be  considered  than  would  otherwise  be  practical.  It  also  provides 
the  potential  for  more  parties  to  participate. 

Questionnaire  conceptualization  and  design  is  difficult.  This 
particular  questionnaire  covers  a complex  and  diverse  subject  and  activ- 
ity. The  questions  have  been  phrased  as  precisely  as  possible  to  avoid 
semantic  difficulties.  To  help  ensure  validity  and  minimize  nonresponse, 
followup  telephone  calls  will  be  made  to  check  for  questionnaire  receipt 
and  to  clarify  any  questions. 

Two  questionnaires  are  being  distributed.  This  questionnaire, 

Type  I,  is  designed  for  the  individual  study.  The  Type  II  questionnaire 
is  concerned  with  studies  and  the  study  process  in  general  at  the  staff 
agency  and  major  command  level  and  is  not  applicable  to  the  individual 
study. 


INSTRUCTIONS 


As  many  questions  as  possible  have  been  designed  to  be  answered  by 
marking  one  or  more  responses  to  each  question.  However,  in  some  instances 


A- 1-2 


ato 


KB 


MO 


« * 


a * 


the  available  choices  may  not  quite  reflect  the  properties  of  the  study 
or  activity  being  characterized.  Respondents  are  requested  to  mark  the 
most  relevant  response,  then  to  make  any  written  commentary  modifying 
the  answer.  Space  for  comments  is  provided  following  each  question. 

Following  certain  questions  there  is  a confidence  scale.  If  you 
are  certain  of  your  answer,  mark  an  "X"  at  the  extreme  right  of  the 
scale.  Please  mark  your  confidence  level  for  all  questions  where  the 
scale  is  indicated. 


0 .1  .2  .3  .4  .5  .6  .7  .8  .9  1 

CONFIDENCE  LEVEL 

On  the  scale  of  0-1,  your  answers  will  be  interpreted  as  follows: 


Virtually  Certain  .9  to  1 

High  .6  to  .9 

Middle  .3  to  .6 

Low  0 to  .3 


Responses  to  some  questions  undoubtedly  will  be  extracted  from 
existing  documents  of  various  kinds.  The  respondent  may  answer  by  refer- 
ence to  a page  and  paragraph  in  the  study  documentation.  For  example, 
rather  than  entering  the  study  objectives,  you  may  reference  the  study 
plan  or  other  document  in  which  the  objectives  are  stated.  THIS  MAY 
ONLY  BE  DONE,  HOWEVER,  IF  THE  STUDY  DOCUMENTS  ARE  APPENDED  TO  THE  COMPLETED 
QUESTIONNAIRE. 
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It  is  important  that  the  name,  office  location,  and  phone  number 
of  the  individual  or  group  completing  this  questionnaire  be  entered. 
Followup  and  feedback  actions  will  flow  through  that  individual  or  group. 
************* 

1.  RESPONDENT  INFORMATION: 

NAME:  

RANK,  TITLE,  POSITION:  

OFFICE  SYMBOL  AND  PHONE  NO:  

ROLE  IN  THIS  STUDY: 


M '* 


2.  STUDY  TITLE: 


(Indicate  the  full  title  and  any  short  titles  or  acronyms) 


3.  The  Army  Study  Program  (TASP)  Category.  Indicate  one  of  the  six  TASP 


categories  for  this  study. 

3-1  MANPOWER  AND  PERSONNEL  _12 

3-2  CONCEPTS  AND  PLANS  25 

3-3  OPERATIONS  AND  FORCE  STRUCTURE  _3 2 

3-4  LOGISTICS 

3-5  SCIENCE  AND  TECHNOLOGY  25 


3-6  MANAGEMENT 


11 


4.  How  was  the  study  accomplished? 


4-1  IN-HOUSE— AD  HOC  28 

4-2  IN-HOUSE— STUDY  AGENCY  56 

4-3  CONTRACT— FCRC  17 

4-4  CONTRACT— NON-FCRC  11 

4-5  PART  IN-HOUSE,  PART  CONTRACT  23 

4-6  OTHER  GOVERNMENT  AGENCY  3 

4-7  OTHER  (SPECIFY  ) 3 


5.  Identify  the  study  agency  or  firm  indicated  in  the  previous  question. 
(If  it  was  an  ad  hoc  group,  specify  the  chairing  organization.) 

NAME: 


6.  Total  Cost.  Care  must  be  taken  to  ensure  that  cumulative  total  costs 
are  reported  if  the  study  spanned  more  than  one  fiscal  year.  State  in-house 
costs  in  professional  man-years  to  the  nearest  tenth.  State  contract 
costs  in  dollars.  Include  costs  for  studies  terminated  prior  to  completion. 

6-1  IN-HOUSE  PMY  626 

6-2  CONTRACT  DOLLARS  $19,527,000 


L- 

0 


-J- 

.1 


_L 

.4 


_L 

.5 


_J X I 1 1 

.6  .7  .8  .9  1 


CONFIDENCE  LEVEL 
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CONFIDENCE  LEVEL 


8.  Duration  of  the  Study. 

8-1  START  DATE 

8-2  COMPLETION/TERMINATION  DATE 


9.  Study  sponsor  and  point  of  contact  (POC)  and,  for  contract  studies 
the  contracting  officer's  representative  (COR). 

AGENCY/COMMAND  

SPONSOR  POC  NAME  

POC  OFFICE  

POC  TITLE/ POSITION  

CONTRACTING  OFFICER  REP 


COR  OFFICE 


10.  Study  Advisory  Group  (SAG).  The  study  sponsor  forms  a SAG  and 
designates  a SAG  chairman  or  designates  a study  manager  if  only  one  agency 
is  involved.  Indicate  the  action(s)  taken  by  the  sponsor  for  this  study. 


10-1  SAG  APPOINTED  63 

10-2  SAG  CHAIRMAN  DESIGNATED  63 

10-3  STEERING  COMMITTEE  FORMED  12 

10-4  STEERING  COMMITTEE  CHAIRMAN  DESIGNATED  12 

10-5  STUDY  MANAGE?  DESIGNATED  30 

10-6  OTHER  MANAGEMENT  TECHNIQUE  USED  36 


(SPECIFY:  Generally  a group  or  committee  similar  to 

a SAG  or  steering  committee.) 

11.  If  a SAG  was  appointed,  list  the  chair  and  member  agencies /commands. 
CHAIR  As  few  as  three  to  as  many  as  20  members  average 
MEMBERS  membership  is  6. 
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12.  Origin  of  Study  Requirement.  The  sponsor  may  not  have  originated  the 
requirement.  For  example,  DOD  or  JCS  may  specify  a study  requirement 
which  is  subsequently  sponsored  by  an  Army  Staff  agency  or  major  command. 


12-1  DA  AGENCY  68. 

12-2  MAJOR  COMMAND  33 

12-3  CSA/SA  21 

12-4  JCS  3 

12-5  DOD  8_ 

12-6  UNSOLICITED  PROPOSAL  4 

12-7  OTHER  4 

OFFICE  IDENTITY: 


(SAUS-OR;  ASD(I&L);  J5;  ODCSOPS;  etc) 


CONFIDENCE  LEVEL 

13.  Indicate  what  type  of  directive,  tasker,  or  action  initiated  the 
study. 


13-1  HQDA  LETTER 

28 

13-2  COMMAND  LETTER 

22 

13-3  DF 

17 

13-4  VERBAL  REQUEST 

19 

13-5  CSM 

14 

13-6  OTHER 

41 

(SPECIFY:  Approved  Contract  23 

Required  by  regulation  or  Mission  18.) 


14.  What  specific  area  or  decision  problem  does  this  study  address? 

Do  not  repeat  the  TASP  category  (question  3) . Choose  up  to  three  areas 
which  best  fit  your  study  from  the  following  list,  indicating  the  order 
of  your  choice.  You  may  add  comments  qualifying  your  choices. 

14-1  SYSTEMS  EFFECTIVENESS  53 
14-2  REQUIREMENTS  (MATERIEL,  FORCES)  47 
14-3  CAPABILITIES  (MATERIEL,  FORCES)  32 
14-4  JUSTIFICATION  OF  PROGRAMS  18 
14-5  DESIGN/PERFORMANCE  EVALUATION  26 
14-6  INPUT  TO  CYCLIC  PLANNING  PROCESS  29 
14-7  ORGANIZATION  EVALUATION/ DEVELOPMENT  17 
14-8  SUPPORT  OF  OPERATIONAL  TESTING  7 
14-9  DOCTRINE  EVALUATION/DEVELOPMENT  12 
14-10  POLICY  DEVELOPMENT/ASSESSMENT  18 
14-11  METHODS  IMPROVEMENT  38 
14-12  THREAT  ANALYSIS  6 
14-13  COST  AND  OP  EFFECTIVENESS  ANALYSIS  20 
14-14  NET  ASSESSMENT  2 
14-15  MODEL  DEVELOPMENT/ IMPROVEMENT  31 
14-16  INPUT  TO  OTHER  STUDIES  26 


14-17  PROGRAM  EVALUATION 


12 


NOTE:  Questions  15-17  may  be  answered  conveniently  by  reference  to  docu- 

ments if  you  append  such  documents  to  the  questionnaire. 

15.  Study  Purpose.  Study  directives,  taskers,  contracts,  and  plans 
include  a statement  of  the  purpose  for  the  undertaking . Indicate  the 
purpose  as  specified  at  the  outset  of  your  study 
PURPOSE: 


16.  Problem  Statement.  Specify  the  original  statement  of  the  problem 
addressed  and  any  restatements  or  refinements  made  in  the  course  of  study 
execution. 

PROBLEM  STATEMENT: 
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17.  specific  Study  Objectives.  Study  objectives  reflect  the  depth  and 
sc„p.  of  attacking  the  ptoblem.  They  establish  goals  for  the  particular 
study  effort.  List  the  objectives  which  were  established  for  your  study. 


OBJECTIVES : 


froken  down  into  1 or  tore  subobjectives  and/or  essential 
_ *1  n f nnalvsis. 


w.  Methodology.  The  methods  to  he  used  in  accomplishing  a study  are 
sometimes  specified  by  directive,  tasker,  or  contract.  For  example, 
a particular  model  (e.g.,  ATLAS) , technique  (e.g.,  random  survey),  or 
analytical  approach  (e.g.,  input-outout)  could  be  specified.  In  other 
cases  the  study  team  selects  a recognised  methodology  or  develops  its 
own.  indicate  which  of  the  following  apply  to  your  study. 

18-1  METHODOLOGY  SPECIFIED  — 28_ 

18-2  EXISTING  METHODOLOGY  USED 
18-3  NEW  METHODOLOGY  DEVELOPED 

" scsusis™  inxt.::.:". 

research  and  analysis. 


106 

41 


34  Checked  more  chan  one  response. 


19.  List  the  names  of  all  models,  techniques,  or  analytical  approaches 
used  in  this  study. 


20.  Data  Sources  and  Validity.  Where  did  the  data  for  this  study  come 
from?  List  the  sources  in  as  complete  a fashion  as  possible.  Describe 
any  checks  which  were  made  to  insure  accuracy,  consistency,  and  overall 
quality  of  the  data. 


DOD  and  Service  Records  and  Files 

Intelligence  Planning  Documents 

Exercise  Plan  of  Analysis 

Official  Documents 

Provided  by  Sponsor 

DPPF,  POM,  JSOP,  etc 

War  Games 

R&D  Results 

Experimental  Data 

No  particular  validation  effort — "are  officially  approved  planning  factors." 
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21.  Number  of  Briefings.  Indicate  the  total  number  to  date  based  on 


this  study. 

21-1  NUMBER  OF  BRIEFINGS:  UNKNOWN  47 

21-2  NUMBER  OF  BRIEFINGS:  (SPECIFY)  94  (Ranged  from  1 to  200; 

average  number  was  9) 


U 

0 


-4- 

.1 


-4- 

.2 


-L- 

.5 


JL 

.6 


-4- 

.7 


.8  .9  1 


CONFIDENCE  LEVEL 


22.  Level  and  Purpose  of  Briefing.  By  level  of  briefing  we  refer  to 
organizational  level.  Typical  purposes  could  be  information,  in-process 
review,  and  decision;  or  you  may  specify  a principal  topic.  List  all 
levels  at  which  this  study  was  briefed  and  indicate  purpose. 


1 1 1 I I I & I I I 1 

0 .1  .2  .3  .4  .5  .6  .7  .8  .9  1 

CONFIDENCE  LEVEL 


PURPOSE  OR 

BY  TO  PRINCIPAL  TOPIC  DATE 


\ 


\ 

) 


Most  were  able  to  indicate  that  briefings  were  given  but 
were  generally  not  sure  of  the  dates  or  the  exact  total  number 
of  briefings. 
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23.  If  your  study  was  terminated  prior  to  completion,  describe  the 
conditions,  circumstances,  and  reasons  for  termination.  Indicate  WHO 
made  the  decision  to  terminate  the  study.  (Complete  the  remainder  of 
the  questionnaire  even  if  your  study  was  terminated.) 


L 


0 .1 


J L 


-L 


_L 


JX L 


.4  .5  .6  .7  .8  .9  1 


CONFIDENCE  LEVEL 


WHO  TERMINATED:  Sponsor,  usually  upon  recommendation  from  SAG  or 

study  manager. 

WHY:  1.  Higher  priority. 

2.  Consolidation  with  another  study. 

3.  Not  a study  but  routine  staff  action  which  is 
a continuous  function. 

4.  Not  a study — falls  under  ADP-AR  18-1. 

5.  Poor  progress. 
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24.  Nature  of  Study  Products.  By  this  is  meant  the  form  of  study  results 
and  physical  products  generated  by  this  study.  Select  as  many  as  are 
applicable  from  the  following  list. 

24-1  STUDY  REPORT  116 

24-2  PROCESS  OR  SYSTEM  FOR  IMPLEMENTATION  29 

24-3  MODEL  FOR  FURTHER  USE  51 

24-4  REFERENCE  DATA  OR  PLANNING  FACTORS  45 

24-5  RECOMMENDATIONS  87 

24-6  INPUT  TO  ANOTHER  STUDY  44 

24-7  INPUT  TO  CYCLIC  PLANNING  PROCESS  22 

24-8  OTHER  (SPECIFY)  Doctrine.  Policies 11 


25.  If  you  have  checked  24-5,  indicate  in  which  of  the  following  general 
»i**as  the  recommendations  pertained. 

25-1  DOCTRINE  29 

25-2  ORGANIZATION  2.8 

25-3  POLICIES  29 

25-4  PROGRAMS  38 

25-5  FORCES  21 

25-6  MATERIEL  39 

25-7  STRATEGY  3_ 

25-8  FOLLOW-ON  STUDY  38 
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26.  Publicity  of  Study  Results.  In  addition  to  briefings  indicated 
in  21  and  22,  which  of  the  following  apply  to  this  study? 


a 


26-1  SUBJECT  OF  SYMPOSIUM  PAPER 


26-2  ABSTRACTED  IN  MILITARY  PERIODICALS 


26-3  CLASSIFICATION  PREVENTED  PUBLICITY 


26-4  PUBLICITY  NOT  SOUGHT 


26-5  ABSTRACTED  IN  DDC,  DLSIE,  ASDIRS 


26-6  REPORT  IN  DDC,  DLSIE,  ASDIRS 


I I I 1 I 1 IX  I I I I 
0 .1  .2  .3  .4  .5  .6  .7  .8  .9  1 


CONFIDENCE  LEVEL 


27.  Distribution  of  Study  Documentation.  This  is  meant  to  include  the 


study  report  and,  if  applicable,  model  and  other  methodology  documentation 


and  user's  guide.  Describe  the  distribution  made  for  your  study.  Identify 


recipients  and  number  of  copies  distributed.  Explain  any  limits  which 


were  placed  on  the  distribution  and  identify  who  imposed  the  limitations. 


(You  may  attach  a copy  of  the  distribution  list.) 


DESCRIBE: 


LIMITS: 


YES  (EXPLAIN)  3 Sponsor  directed  no  distribution. 


138  Other  than  security  classification. 
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28.  Study  Achievements.  Referring  to  the  objectives  of  this  study, 
choose  the  one  statement  which  in  your  opinion  best  describes  the  study 
results.  Consider  "most"  to  mean  more  than  half. 

28-1  NO  OBJECTIVES  JET  9 

28-2  ALL  OBJECTIVES  MET  77 

28-3  MOST  OBJECTIVES  MET  42 

28-4  MOST  OBJECTIVES  NOT  MET  13 


0 .1  .2  .3  .4  .5  .6  .7  .8  .9  1 


CONFIDENCE  LEVEL 


A- 1-17 


29.  Now  describe  as  best  you  can  the  reasons  why  objectives  were  not 


met  or  only  partially  met.  Be  as  specific  as  possible.  If  all  objectives 
of  this  study  were  met,  go  to  the  next  question. 


DESCRIBE: 


"Insufficient  time  and  people  to  do  everything." 

"DA  Staff  reorganization." 

"CONUS  reorganization." 

"Objectives  were  dropped  by  sponsor." 

"Methodology  not  suitable." 

"Approach  did  not  adequately  address  all  objectives." 


0 .1  .2  .3  .4  .5  .6  .7  .6  .9  1 


CONFIDENCE  LEVEL 
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30.  Approvals,  Disapprovals,  and  Staffing.  Describe  the  channels 
through  which  the  results  of  this  study  were  staffed  to  obtain  approval 


and  concurrence.  Indicate  the  organization,  level,  and  title  or  position 
of  those  who  could  approve  or  disapprove  the  final  recommendations. 
DESCRIBE: 


Usual  chain  of  command,  sponsor,  and  SAG/steering  group 
channels . 


31.  Approvals,  Disapprovals,  and  Staffing.  This  question  is  similar  to 
the  previous  question.  In  this  case  indicate  what  types  of  decisions  were 
made  (i.e. , approval,  disapproval,  concurrence,  or  nonconcurrence). 

Specify  the  title  and/or  position  of  the  decisionmaker  and  the  date  of 
the  decision. 

SPECIFY : WHO  TYPE  WHEN 


Most  found  it  difficult  to  reconstruct  these  events. 


0 .1  .2  .3  .4  .5  .6  .7  .8  .9  1 


CONFIDENCE  LEVEL 
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32.  Based  upon  the  information  available  to  you,  list  all  agencies. 


commands,  or  other  activities  that  have  or  are  using  the  results  of  this 
study . 


0 .1  .2  .3  .4  .5  .6  .7  .8  .9  1 


CONFIDENCE  LEVEL 


LIST: 


Except  for  their  own  agency /command,  few  were  confident 
in  identifying  others  who  used  or  are  using  particular  study 
results. 


33.  Are  there  any  agencies,  commands,  or  other  activities  that  you  feel 
could  use  the  results  of  this  study  but  who,  to  the  best  of  your  knowledge, 
do  not? 


33-1  NO 


33-2  YES  (LIST) 


83 (only  111  answered 

this  question) 

28 


0 .1  .2  .3  .4  .5  .6  .7  .8  .9  1 


CONFIDENCE  LEVEL 
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34.  Describe  how  the  results  of  this  study  are  being  used  or  could  be 
used  by  those  you  have  identified  in  32  and  33.  Cite  a "for  instance" 
for  each  different  use. 


.J, 


DESCRIBE: 


Uses  were  identified  in  relation  to  the  purposes  of  the 
particular  study.  Responses  were  categorized  as  follows: 


Direct  input  to  decision/problem  solution  59 
Input  to  follow-on  study  20 
Input  to  planning  systems  or  processes  24 
Reference  data  or  planning  factors  19 
Not  used  19 


35.  Benefits  From  This  Study.  Describe  the  benefits  that  vou  feel  were 
realized  because  this  study  was  done.  Include  direct  and  indirect  benefits 
Indirect  benefits  are  such  as  professional  development,  training  of 
study  participants,  unintended  identification  of  other  problem  areas, 
creation  of  contingency  study  capability,  and  methodology  state-of-the-art 
advancement.  Indicate  which  benefits  are  auditable.  Describe  the  benefits 
and  recipients  as  precisely  as  possible. 

DESCRIBE: 


Respondents  had  considerable  difficulty  in  identification 
of  benefits  in  precise  terras.  Phrases  such  as  significant, 
widespread,  Army-wide,  etc  were  used.  Only  eleven  thought 
benefits  were  "auditable."  Indirect  benefits  most  frequently 
mentioned  were  related  to  increased  knowledge  and  capabilities 
at  all  levels  (study  agency  and  staff)  to  approach  and  solve 
problems  whether  they  are  studies  or  not. 
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36.  Measure  of  Benefits.  Specify  what  you  regard  as  a reasonable 
measure  of  the  benefits  and  success  of  this  study.  Here  you  are  asked 
to  attach  a value  to  the  benefits. 

SPECIFY : 


Qualitative  Description:  AO 


Dollar  Value  Indicated:  22 


Cannot  Estimate: 


37.  Lnfluence  on  Decisions.  Do  you  know  of  any  decisions  that  were  made 
using  the  results  of  this  study  as  input  (directly  or  indirectly)  to  the 
decisionmaking  process?  List  and  describe  these  decisions. 


0 .1  .2  .3  .A  .5  .6  .7  .8  .9  1 


CONFIDENCE  LEVEL 


77  responded  that  studies  were  used  directly  or  indirectly 
in  some  decision. 

A9  stated  "Unknown." 

15  stated  "Not  Applicable." 
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J8.  In  your  opinion  what  is  the  most  important  decision  or  problem 
solution  that  this  study  has  influenced? 

SPECJFY: 


93  responded  to  this  question,  69  indicating  a specific 
decision  (direct  or  indirect);  the  remainder  specified  a 
general  problem  area  (i.e.,  force  structuring)  where  results 
were  used. 


39.  Knowing  what  you  know  now  and  if  this  study  was  just  beginning, 
what  things  do  you  feel  should  be  done  differently  to  improve  its 
success  and  usefulness? 

LIST: 


"More  time . ' 

"Better  planning." 

"Start  sooner." 

"Get  command  interest  early-on." 
"Nothing. " 


I 
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AO.  What  would  have  been  the  impact  if  this  study  had  not  been  done? 
Describe  as  best  you  can  the  nature  and  extent  of  the  impact. 


I 1 1 U 
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CONFIDENCE  LEVEL 


No  Impact:  19 
Low  Impact : 35 
Significant  Impact:  78 
Unknown : 9 


Al.  You  are  asked  to  provide  copies  of  all  pertinent  literature  from 
the  files  for  this  study.  Listed  below  are  the  items  we  are  most 
interested  in  obtaining.  Should  you  be  unable  to  forward  them  with  this 
completed  questionnaire,  indicate  whether  they  exist  or  not,  their  loca- 
tion, classification,  and  explain  how  we  must  proceed  t"  obtain  them. 
Al-1  DIRECTIVE,  TASKER,  CONTRACT 
A 1-2  RACS 
A 1-3  STUDY  PLAN 
Al-A  BRIEFING  SCRIPTS 
A 1-5  SAG  MEETING  MINUTES 
Al-6  STUDY  REPORT 
Al— 7 MODEL/METHOD  DOCUMENTATION 
Al-8  APPROVAL/ IMPLEMENTATION  DOCUMENTS 
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QUESTIONNAIRE:  ARMY  STUDIES — RESULTS , USES,  AND  BENEFITS 

TYPE  II 

INTRODUCTION 


The  purpose  of  this  questionnaire  is  to  aid  in  gathering  data  for 
use  in  evaluating  the  results,  uses,  and  benefits  of  Army  studies.  Use 
of  this  questionnaire  makes  it  possible  for  more  parties  to  participate. 

It  permits  members  of  the  Army  study  community  to  describe,  characterize, 
and  assess  their  own  study  activities. 

Questionnaire  conceptualization  and  design  is  difficult.  This 
particular  questionnaire  covers  a complex  and  diverse  subject  and  activity. 
The  questions  have  been  phrased  as  precisely  as  possible  to  avoid  semantic 
difficulties.  To  help  ensure  validity  and  minimize  nonresponse,  followup 
telephone  calls  will  be  made  to  check  for  questionnaire  receipt  and 
clarify  any  questions. 

Two  questionnaires  have  been  distributed.  This  questionnaire, 

Type  II,  is  concerned  with  studies  and  the  study  process  in  general  at 
the  staff  agency  and  major  command  level.  Type  I is  designed  for  the 
individual  study  and  those  involved  with  a particular  study. 

INSTRUCTIONS 

As  many  questions  as  possible  have  been  designed  to  be  answered  by 
marking  one  or  more  responses  to  each  question.  However,  in  some 


! 


instances  the  available  choices  may  not  quite  reflect  the  properties 
of  the  subject  or  activity  being  characterized.  Respondents  are  requested 
to  mark  the  most  relevant  response,  then  to  make  any  written  commentary 
modifying  the  answer.  Space  for  comments  is  provided  following  each 
question. 

Following  certain  questions  there  is  a confidence  scale.  If  you 
are  certain  of  your  answer  mark  an  "X"  at  the  extreme  right  on  the  scale. 
Please  mark  your  confidence  level  for  all  questions  where  the  scale  is 
indicated. 
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CONFIDENCE  LEVEL 

On  ' e scale  of  0-1,  your  answers  will  be  interpreted  as  follows: 
Virtually  Certain  .9  to  1 

High  .6  to  .9 

Middle  .3  to  .6 

Low  0 to  .3 

Responses  to  some  questions  undoubtedly  could  be  extracted  from 
existing  documents  of  various  kinds.  Respondents  may  answer  such 
questions  by  reference  to  the  documents  PROVIDING  the  documents  are 
appended  to  the  completed  questionnaire. 
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RESULTS  AND  USE  OF  ARMY  STUDIES 


I.  GENERAL 


1.  Purpose.  This  study  assesses  the  results,  uses,  and  benefits  of 
Army  studies  to  be  considered  by  the  Director  of  Management,  Office  of 
the  Chief  of  Staff  of  the  Army  (OCSA) , in  decisions  and  actions  regarding 
the  directions  of  current  and  future  Army  study  efforts. 

2.  Scope. 

a.  This  report  is  based  on  analysis  of  a 145-study  sample 
selected  from  over  460  studies  completed  or  terminated  during  FY  74  and 
FY  75.  In-house  and  contractor  studies  in  all  The  Army  Study  Program 
(TASP)  categories  have  been  included. 

b.  The  evaluation  of  each  study  did  not  include  a peer  review 
of  study  contents  or  procedure  or  any  attempt  to  duplicate  the  results. 
Rather,  the  analysis  concentrated  on  results  of  the  studies  in  terms  of 
the  objectives  and  the  uses  made  of  these  results.  Mailed  questionnaires 
served  as  the  primary  data-collection  instrument. 

c.  Both  tangible  and  intangible  benefits  are  assessed  relative 
to  study  purposes,  issues  addressed,  achievements,  and  uses  in  the  limited 
context  of  individual  study  evaluation.  The  much  broader  question  of 


whether  individual  study  goals  and  objectives  are  appropriate  is  consid- 
ered a part  of  total  program  evaluation  and  is  given  much  less  attention 
in  this  study.  (Howevei  , insights  related  to  program  evaluation  have 
been  supplied  to  the  Director  of  Management  in  a separate,  informal  paper.) 


u 


It  is  also  important  that  the  name,  office  location,  and  phone 


number  of  the  individual  or  group  completing  the  questionnaire  be 
entered.  Followup  and  feedback  actions  will  flow  through  that  individual 
or  group. 

************** 

1.  RESPONDENTS  INFORMATION: 

NAME:  

RANK,  TITLE,  POSITION:  

AGENCY/COMMAND:  

OFFICE  SYMBOL  AND  PHONE  NO:  


2.  Length  of  time  you  have  been  working  with  or  a part  of  the  Army 
study  system. 

More  than  7 years — 4;  4 to  7 years — 2;  2 to  4 years — 2; 

1 to  2 years — 3;  Less  than  1 year — 3. 

3.  Study  Coordinators  serve  as  their  agency  or  command  point  of  contact 
for  outside  agencies  on  all  study  matters.  List  the  information  concern- 
ing your  agency/ command  and  study  coordinator. 

STUDY  COORDINATOR  NAME:  

PHONE  NO:  

OFFICE  SYMBOL:  
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4.  There  are  six  categories  of  Army  studies.  What  proportion  of  your 
total  agency/ command  study  resources  expended  during  FY  74  and  FY  75 
was  devoted  to  each?  Develop  proportions  using  professional  man-years 
(PMY)  for  in-house  and  dollars  for  contract. 

PROPORTION  OF  RESOURCES: 


In-house  Contract 


FY  74 

FY  75 

FY  74 

FY  75 

4-1 

MANPOWER  AND  PERSONNEL 

11.5% 

10.5% 

10.1% 

10.3% 

4-2 

CONCEPTS  AND  PLANS 

20.3% 

18.6% 

22.5% 

17.7% 

4-3 

OPERATIONS  AND  FORCE  STRUCTURE 

24.8% 

26.2% 

24.1% 

21.1% 

4-4 

LOGISTICS 

23.6% 

21.9% 

18.9% 

19.2% 

4-5 

SCIENCE  AND  TECHNOLOGY 

9.6% 

10.0% 

11.4% 

11.7% 

4-6 

MANAGEMENT 

10.2% 

12.8% 

13.0% 

20.0% 

100.0% 

100.0% 

100.0% 

100.0% 

i i i i i l 1 

1 

1 x 1 1 
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5.  This  question  is  similar  to  the  previous  question.  In  this  case 
list  the  proportion  of  studies  accomplished  by  in-house  or  contract 
means  by  study  category. 

PROPORTION  OF  STUDIES: 


In-house 


Contract 


1 

i 

FY  74 

FY  75 

FY  74 

FY  75 

5-1 

MANPOWER  AND  PERSONNEL 

71  % 

63  % 

29  % 

37  % 

i 

I 5-2 

CONCEPTS  AND  PLANS 

68  % 

72  % 

32  % 

28  % 

5-3 

OPERATIONS  AND  FORCE  STRUCTURE 

57  % 

64  / 

43  % 

36  % 

5-4 

LOGISTICS 

72  % 

83  % 

28  % 

17  % 

5-5 

SCIENCE  AND  TECHNOLOGY 

59  % 

66  % 

41  % 

34  % 

5-6 

MANAGEMENT 

70  % 

71  % 

30  % 

29  % 
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CONFIDENCE  LEVEL 
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6.  There  are  several  ways  to  conduct  a study.  Indicate  those  your 


agency /command  has  used  in  FY  74  and  FY  75 

and  is  currently  using. 

FY  74 

FY  75 

CURRENT 

6-1  AGENCY/ COMMAND  STUDY  AGENCY 

8 

9 

9 

6-2  AGENCY/ COMMAND  AD  HOC  GROUP 

10 

. 9 

9 

6-3  ARMY  STUDY  AGENCY  (NOT  YOURS) 

7 

. 10 

9 

6-4  CONTRACT 

10 

9 

8 

6-5  OTHER  (SPECIFY) 

0 

0 

0 

1 1 ! 1 1 1 1 

1 1 lx 

1 

0 .1  .2  .3  .4  .5  .6 

.7  .8  .9 
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7.  Development  of  your  annual  study  program: 

[Briefly  describe  how  your  agency /command  annual  study  requirements 
are  identified  and  put  together  as  a study  program.  This  could  be  done 

v 

on  a separate  sheet  or  if  you  have  a directive  or  memo,  simply  attach 
a copy.] 

All  respondents  identified  procedures  keyed  to  the  pro- 
visions of  AR  5-5  and/or  the  PPBS.  Most  had  formally  pub- 
lished Staff  memoranda,  taskers,  or  supplements  to  AR  5-5 
assigning  responsibilities  for  their  study  program  develop- 
ment and  execution. 
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8.  Agencies/commands  often  conduct  studies  for  other  sponsors.  Indicate 
the  extent  to  which  your  agency/command  conducts  studies  sponsored  by 


others. 


FY  74  FY  75  CURRENT 


8-1  MORE  THAN  75%  OF  STUDIES 


8-2  50-75%  OF  STUDIES 


8-3  25-50%  OF  STUDIES 


8-4  0-25%  OF  STUDIES 


8-5  NEVER 
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CONFIDENCE  LEVEL 


9.  Indicate  the  degree  to  which  the  annual  Study  Planning  Guidance 
influences  the  formulation  of  your  agency/command  study  program. 

9-1  DECISIVE  2 

9-2  IMPORTANT  4 


9-3  MARGINAL 


9-4  NO  INFLUENCE 


I I I I I I I X I I II 
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10.  What  is  the  single  most  important  factor  in  determining  the  content 
of  your  agency/command  study  program? 

10-1  AGENCY/ COMMAND  MISSION  AND  FUNCTIONS  9 

10-2  PLANNING  DOCUMENTS  (DPPG,  POM,  ETC.)  1 

10-3  CSA  STUDY  PLANNING  GUIDANCE  2 

10-4  OTHER  (SPECIFY)  2 


( Execut 


Command  Guidance 
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CONFIDENCE  LEVEL 

11.  Where  do  you  expect  the  results  of  your  agency /command  study 
program  are  most  frequently  used? 

11-1  INTERNAL  (YOUR  AGENCY/COMMAND) 


13-2  EXTERNAL  AGENCY /COMMAND  (WITHIN  ARMY) 
11-3  EXTERNAL  ARMY  (BY  OSD,  JCS) 
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12.  Describe  your  FY  74,  FY  75,  and  current  study  program  in  terms  of 
resources  (in-house  professional  man-years,  dollars  contract). 


EXPENDED  PROGRAMED 


FY  74 

FY  75 

CURRENT 

12-1 

IN-HOUSE  (PMY) 

- 9.?4.,i£. 

1.265.9 

12-2 

CONTRACT  ($K)  (TOTAL) 

6.280.70 

9.265.0 

OMA  ($K) 

638.0 

UMj-ZQ 

2.555.0 

RDTE  ($K) 

5,934.5 

5.0Q4.QQ 

6.710.0 

OTHER  ($K) 

132.00 

1 I 1 1 1 1 

1 1 XI 

1 1 
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13.  When  study  requirements  exceed  available  resources,  decisions  are 
made  concerning  requirements  which  will  not  be  met.  Usually  a system 
of  priorities  is  employed.  Describe  how  priorities  are  assigned  in 
your  agency/command . 


Responses  included  examples  of  formally  documented  priority 
scoring  systems  as  well  as  general  policy  statements.  Over- 
all, the  schemes  establish  priorities  in  the  following  order: 

1.  Higher  directives. 

2.  Executive,  Commander  emphasis. 

3.  Urgent  mitsion  and  function  requirements. 

4.  Overall  guidance  (SPG,  POM,  DPPG) . 
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14.  How  frequently  is  your  study  program  progress  reviewed  internally? 


14-1  EVERY  30  DAYS  OR  LESS 


14-2  QUARTERLY 


14-3  SEMI-ANNUALLY 


14-4  ANNUALLY 


14-5  NO  REGULAR  REVIEW 


15.  Completed  studies  may  or  may  not  be  approved  by  the  sponsor.  For 
those  studies  sponsored  by  your  agency /command  and  completed  during 
FY  74  and  FY  75,  indicate  the  proportion  of  studies  in  the  listed 
categories . 


15-1  APPROVED  FOR  AGENCY /COMMAND  USE  ONLY 


15-2  APPROVED  FOR  INTERNAL  AND  EXTERNAL  USE 


FY  74  FY  75 


17.1%  IS. 3% 
63.1%  62.4% 


15-3  PARTIALLY  APPROVED 


5.0%  6.2% 


15-4  NOT  APPROVED 


2.4%  2.1% 


15-5  SPONSOR  APPROVAL  NOT  REQUIRED 


4.0%  3.6% 


15-6  APPROVAL  REQUIRED  NEXT  HIGHER  HEADQUARTERS  5.7%  4.3% 


15-7  UNKNOWN 


2.7%  3.1% 
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16.  Efforts  have  been  made  to  facilitate  the  exchange  of  information 
among  all  participants  in  the  Army  Study  Program.  Examples  include  the 
use  of  study  information  data  banks;  progress  reporting  procedures; 
and  exchange  of  agency /command  study  programs.  List  the  sources  or 
procedures  that  your  agency /command  routinely  uses  to  obtain  information 
about  ongoing,  completed,  or  planned  study  efforts. 


DDC 

DLSIE 

ASDIRS 


Study  Management  Office 
Exchange  of  Study  Program 
Personal  Contact 
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17.  Studies  are  terminated  before  completion  for  a variety  of  reasons. 
Order  the  following  from  the  most  frequent,  1,  to  least  frequent,  7, 
cause  for  termination  in  your  agency /command.  (Note:  Order  those  which 

apply.  Enter  a zero  for  those  which  are  not  a cause  for  termination  in 
your  agency /command.)  (One  response  was  all  Zeros.) 

1 2 3 4 5 6 7 0 


17-1  CONTRACT  FUNDS  LIMITATIONS 


110  110  1 


17-2  DUPLICATE  ANOTHER  STUDY 


0000010  13 


17-3  UNSATISFACTORY  PROGRESS 


0120010  10 


n 

I } 

|1|  wH 


17-4  PROBLEM  PASSED  WITH  TIME 


17-5  PREEMPTED  BY  HIGHER  PRIORITY 


3 2 2 1 0 1 1 

8 3 2 0 0 0 0 


17-6  TECHNICAL/METHODOLOGY  PROBLEMS  2312110 


17-7  OTHER  (SPECIFY  * 


1110000  11 


Middle 
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*Change  in  management/action  officer  with  resultant  change  in  emphasis. 
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18.  It  is  extremely  difficult  to  measure  and  ascribe  a value  to  the 
results  of  an  .agency/ command  study  program.  It  is  often  just  as  difficult 
to  specify  the  value  or  benefit  of  an  individual  study.  Considering 
the  resources  that  your  agency/ command  commits  to  its  annual  study 
program,  select  the  category  below  which  best  describes  the  program's 
cost  effectiveness. 


18-1  HIGHLY  COST  EFFECTIVE 


18-2  COST  EFFECTIVE 


18-3  NOT  COST  EFFECTIVE 


18-4  UNKNOWN  (NO  OPINION) 
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19.  Referring  to  your  response  to  the  previous  question,  describe  the 
nature  of  the  effectiveness  or  benefits  resulting  from  your  study 
program.  Include  tangible  and  intangible  results  that  you  feel 
benefit  your  agency/command  or  the  Army  even  if  they  do  not  fit  neatly 
in  a cost-eff.*ctiveness  equation.  Cite  concrete  examples  where  possible 
and  the  extent  of  the  benefit/effect;  i.e.,  internal,  Army-wide,  DOD- 
wide.  (This  question  can  be  answered  on  separate  sheets  of  paper.)  If 
you  choose  response  18-4,  explain  why  you  are  unable  to  form  an  opinion. 

In  general,  the  responses  tc  this  question  were  in  terms 
of  fulfilling  mission  and  function  responsibilities  relative 
to  the  current  Army  posture  and  meeting  PPBS  requirements. 
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20.  Considering  your  responses  to  the  two  previous  questions,  can  you 
estimate  a benefit/cost  ratio  for  your  total  study  program?  Briefly 
explain  your  choice. 

20-1  YES  3 : ESTIMATED  RATIO  1 ; 1 

20-2  NO  11 


U 

0 


l 


.1 


.2 


.3 


-L 

.4 


-L- 

.5 


jJLi 


.8  .9 
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EXPLAIN: 


Yes  responses  were  all  qualitative. 

Explanation  of  no  responses  centered  around  the  extreme 
difficulty  of  measuring  benefits  in  concrete  terms.  All  pointed 
out  that  some  studies  lend  themselves  to  cost/benefit  analysis 
but  most,  and  the  program  as  a whole,  do  not. 
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21.  Based  on  your  agency/command  experiences,  how  do  your  In-house 
study  sources  compare  with  contract  study  sources  in  terms  of  effective- 
ness? Briefly  describe  why,  citing  one  or  more  "for  instances." 

21-1  EQUAL  8 

21-2  CONTRACT  BETTER  

21-3  IN-HOUSE  BETTER  6 

I I I I I I I I I IX— J 
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WHY: 


"More  responsive,  quicker,  and  probably  more  thorough 
if  done  in-house." 

"Only  advantage  of  contractor  is  he  doesn't  have  an 
axe  to  grind." 

"Contractors  generally  do  not  have  the  same  depth  of 
military  oriented  problem  experience . " 

"In-house  sources  really  excel  if  given  the  necessary 
priorities  and  dedicated  guidance." 
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Regardless  of  how  well  we  are  doing  in  the  study  business , 
most  will  agree  that  there  is  always  room  for  improvement.  List  in 
order  of  importance  those  actions  (or  inactions)  that  you  feel  would 
do  the  most  to  improve  the  success  and  usefulness  of  the  Army  Study 
Program. 

LIST: 


"Reduce  the  Administrative  burden." 

"Need  access  instructions  to  DA  studies  data  base." 

"Improve  planning  guidance.  Current  guidance  is  too 
burdensome  to  be  useful." 

"Improve  accuracy  of  data  bank  information  by  quality 
control  of  data  input." 

"There  ought  to  be  a better  coordination  policy." 

"The  paperwork  needs  to  be  tailored,  that  is,  should 
be  reduced." 

"Currently,  the  system  suffers  from  a lack  of  high 
level  direction  that  it  was  originally  designed  to 
have." 

"Demand  quality  within  our  study  agencies.  Do  not 
trade  off  quality  and  substance  for  form  of  the 
program  or  effort." 

"Do  better  program  planning  to  reduce  fluctuations 
and  turbulence." 

"Increased  communications  between  DA  study  office 
and  study  coordinators.  Increased  meetings  of 
study  coordinators." 
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23.  Respondent  commentary.  You  may  add  comments  relative  to  studies, 
the  Army  Study  System,  the  merits  of  this  endeavor,  or  any  pertinent 
subject. 


"...hope  the  information  retrieved  by  questionnaire  will 
give  the  credibility  inherent  to  a team  independently 
gathering  data.  1 feel  a questionnaire  allows  too  much 
room  for  individual  interpretation  of  some  of  the  facts." 

"This  particular  effort  to  further  determine  the  costs 
and  benefits  of  the  Army  Study  System  is  worthwhile, 
though  frustrating.  It  highlights  a need  by  study 
agencies  to  document  results  and  uses  of  studies — a 
necessary  endeavor  if  the  study  system  is  to  continue." 
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I.  STRATIFIED  RANDOM  SAMPLE 


Category  1 — In-house 

Army  Direct  Support  (SIGNIT)  Resources 

Study  of  the  Army  Publication  System  (STARPUBS) 

Processing  of  Words  Efficiently  and  Effectively  (POWER) 

Professional  Accredidation  Program  for  Law  Enforcement  Personnel 
(Career  Management  Field  95) 

Category  1 — Contract 

Analytic  Techniques  for  Examination  of  Manpower  Policy  Alternatives 
Enlisted  Personnel  Assignment  System  Study 

Category  2 — In-house 
Cost  Effectiveness  Study  on  DAAL 
Tactical  Nuclear  Weapons  Requirements  Methodology 
Lance  Missile  Battery  Red  Team  Analysis 
Carmonette  Night  Vision  Analysis 

Integrated  Battlefield  Control  System — Command  and  Control  Concept  for 
Echelons  Above  Division  (IBCS/EAD) 

Nuclear  Force  Posture — An  Analysis  of  Selected  Alternatives  (NUCFO) 
Weapons  Requirements  to  Support  Nuclear  Targeting  Options 
Nuclear  Doctrine,  Organization,  and  Equipment  (NUDORE  I) 
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Category  2 — Contract 


Long  Range  Standardization  of  Air  Traffic  Control  Equipment 

Automated  Technical  Control  Information  Brochure  and  Staff  Planners 
Guide 

Net  Technical  Assessment  of  Proving  Ground  and  Test  Facilities 

Methodology  for  Integrated  Forces  Requirements  and  Capabilities 
(MINTCAP) 

NATO  Combat  Capabilities  Analysis  (COMCAP  III) 

Category  3 — In-house 
TOV) /DRAGON  Survivability  Study 

Mutual  and  Balanced  Force  Reduction  Wargames  and  Analyses  (MBFR  II) 
Mine  Plow  Evaluation  Study 

Conceptual  Designs  of  the  Army  in  the  Field  (CONAF)  III 
Analysis  of  the  Army  Requirement  to  Own  and  Operate  Watercraft 
Cavalry/Scout  Study 
Force  Stratification  Analysis 

Joint  OSD/DA  NATO  Land  Forces  Requirements  Review 
Army  Total  Force  Study — 1974 
Division  SHORAD  Study  (DIVAD) 

Requirements  Methodology  (TAN  REM)  Phase  II 
Tank  Special  Study  Group  (TSSG) 


Category  3 — Contract 


Tactical  Effectiveness  Testing  Antitank  Missiles  (TETAM) 
Wideband  Communications  at  Unused  Frequencies 
Modification  of  Math  Model  for  Small  Independent  Action 
Development  and  Improvement  of  CEM 
Force  Structure  and  Programing  (SPECIFOR) 

GLOBAL  Model  Documentation 

Rough  Notes  on  Ways  of  Improving  US/NATO  Antitank  Capability 

Category  4 — In-house 
Tactical  Landing  System  (COEA) 

Interrelationship  of  Management  Indicators 
Feasibility  of  Eliminating  Depot  Maintenance  in  USAREUR 
Army  Replacement  Requirements  Objectives  for  Wartime 
U.S.  Army  Trans-Hydro  Craft  (TRANS-HYDRO)  1975-1985 
Programed  Review  of  Basic  Elements  (PROBE) 

The  Army  in  the  Field  Container  System  Study 

Contaminated  Area  Clearance  and  Land  Use  Alternatives  (CLEAR) 

Ammunition  Fire  Fighting  Doctrine 

Army  Retail  Materiel  Management  Model  (ARMMM) 

Aircraft  Refueling  and  Rearming  System  (ARRS) 
aN/URC-78  Cost  Effectiveness  Study 
Army  Installation  Energy  Requirements  in  CONUS 
Management  of  Army  Wholesale  Logistics  Literature  (AWLL) 
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Category  4 — Contract 


Operational  Readiness  Policies  for  Selected  Units  and  Weapons  Systems 
Evaluating  and  Improving  the  Direct  Support  System  Subelements 
Improved  Methods  for  Developing  Army  Worldwide  Asset  Position 
Logistics  Performance  Standards  (Phase  II) 

Tactical  Vehicle  Fleet  Inventory  Model 

Category  5 — In-house 

Personnel  Armor  System  for  Ground  Troops  (COEA) 

Reevaluation  of  60%  Gradability  Requirement 
Universal  Drivers  Viewer 

Remotely  Piloted  Aerial  Observer/Designator  System  (RPAODS) 

Study  of  the  Susceptibility  of  SAM-D  to  ARM  Attack 

Initial  Evaluation — FRG  vs  U.S.  Track  Design 

Probability  of  Hits  of  Aircraft  by  Antiaircraft  Fire  (HITVAL) 

Deadline  Cost  Model  Study 

A Study  of  the  Benefit/Risk  Involved  in  Replacing  Current  Explosive 
Melt-Pour  Facilities 

Red  Team  Evaluation  of  Transfer  vs  Convention  Machines 
Loop  Optimization 

Automatic  Steerable  Null  Antenna  Processors 
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II.  CONTRACT  STUDY  ADD-ONS— COSTS  MORE  THAN  $300K 


Army  Manpower  Prediction  System  (AMPS)  (Part  II) 

Methodology  for  Conventional  Purpose  Forces  Under  the  Total  Force 
Concept  (METOFOR  II)— FY  74 

Impact  of  Alternative  National  Strategies  on  Army  Planning 
Nike-Hercules  System  Simulation  Model 

Analytic  Support  for  Reserve  Component  Field  Experimentations  and 
Evaluations 

Evaluation  jf  Reserve  Component  Improvement  Concepts  (Cost  and 
Effectiveness) 

Fast  Frequency  Hopping 

Systems  Improvements  to  the  Reserve  Components  Personnel  Projection 
Model  (RP2M)  (II) 

Evaluation  of  Modern  Volunteer  Army  Program  (Phase  III) 

Family  of  Observation,  Scout,  and  Attack  Helicopters  (SCAT  II) 

IBCS  Division  Level  Systems  Definition,  Staff  Organization  and 
Procedures,  3d  Refinement  (IBCS-3D  REF) 

Integrated  Global  Force  Posture  Analysis  (Task  Order  1) 

Technical  Assistance  to  the  DSS  Task  Group 

Standard  Army  Management  Language,  Phase  II  (SAML  II) 

Computer  Programming  Support  for  Conversion  of  Models  and  Simulations 

Air  and  Ballistic  Missile  Defense 

Models  of  the  U.  S.  Army  Worldwide  Logistics  System  (MAWLOGS) 
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III.  IN-HOUSE  STUDY  ADD-ONS — COSTS  MORE  THAN  6 PMY 


Education  of  Army  Officers  Under  the  Officer  Personnel  Management 
System. 

Records  Administration  in  Microform  Mode  (RAM  2) 

Production  Base  Plan  for  the  FY  76  Planning  Period 
Force  Assessment  and  Capability  Evaluation  (FACE) 

Op  Effectiveness  Analysis  (HLH  COEA) 

Dragon  Cost  and  Op  Effectiveness  Analysis  (DRAGON-COEA) 

CEM-ATLAS  Wargame  Comparison  (WAGCOM) 

Armored  Reconnaissance  Scout  Vehicle  Study 
Integrated  Support  Services  MIS  (ISSMIS) 

DSU  Repair  Parts  Stockouts 

Logistic  Scenario  Oriented  Recurring  Evaluation 

Economic  Modeling  of  Army  Ammunition  Production  Base 

Army  Qualitative  Resource  Requirements  for  Nuclear  Weapons  Effects 
Information 

Image  Simulation  (Thermal/Intensifier) 
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IV.  AGENCY/COMMAND  NOMINATIONS 


Repair  Parts  Study 

US  Army  Aircraft  Peacetime  Replacement  Factors 

Track  Cost  Model 

AVSCOM  Measures  of  Effectiveness 

Criteria  for  Establishing  Planning  Factors 

Evaluation  of  Operations  and  Programing  Effectiveness  for  Mechanized 
Stock  Control  Processing 

Financing  of  Army  Inventory 

USACC  EMP  Program  Requirements 

Supply  and  Maintenance  Support  Concept  for  USACC 

AL'REP 

STRAMS  Update 

32H  Series  Communications  Study  for  FY  74-76 
USASA  Tactical  Communications  Study  FY  76-86 
Priorities  for  Allocation  of  COMSEC  Resources 
War  Reserves 

Review  and  Analysis  of  the  Human  Self-Development  Program — FY  75 
Wartime  Active  Replacement  Factor  Study  (Phase  III) 

Evaluation  of  TRICAP  Division 
Company  Administration  Study 
Air  Movement  Planning  System 

Nuclear  Cannon  Projectile  Study,  Phase  II,  155  mm — Artillery  Fired 
Projectile  (COEA) 


Accuracy  Analysis-  of  Arty  Cannon  Systems 
IBCS  Concept  for  ATACOMAP 

Organizational  Development  Pilot  Test  for  Army  Personnel  Center 
People  Management 

Student  Instructor  L .d  Model  (SIL  III) 

Contingency  Planning  and  Forecasting  (FORECAST  90) 

NIKE  HERCULES  Effectiveness  Study  (1976-1980)  (NIKE  77) 

Nonnuclear  Ammunition  Combat  Rates  Programing  FY  76-80 

An  Analysis  of  Deployment  of  the  101st  Airbo’  -<e  Division  (Air  Assault) 
to  Europe 

Management  Study  on  Housekeeping  Service  for  Dwight  D.  Eisenhower  Army 
Medical  Center 

Evaluation  of  Expar  ed  Role  of  Physical  Therapist  in  Screening 
Musculoskeletal  Disorders 
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